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COMPARISON  OF  DIRECT  INSTRUCTION  AND  PROBLEM  CENTERED  INSTRUCTION 
FOR  ARMY  INSTITUTIONAL  TRAINING 

EXECUTIVE  SUMMARY 


Research  Requirement: 

One  tenet  of  the  Army  Learning  Model  (ALM)  for  2015  (U.S.  Army,  2011)  is  that 
institutions  transform  existing  instruction  and  develop  new  “context-based,  collaborative, 
problem-centered  instruction  (PCI).’’  There  is  no  guarantee  that  in  all  cases  a  pivot  to  PCI  will 
result  in  more  effective  training  (cf.  Kirschner,  Sweller  &  Clark,  2006;  Mayer,  2004)  or  that  any 
gains  due  to  PCI  will  justify  the  effort  involved  in  making  the  pivot.  Further,  there  is  evidence 
that  PCI  is  more  suited  to  ill-defined  subject  domains,  while  direct  instruction  (DI)  is  more  suited 
to  well-defined  subject  domains  (Tobias  and  Duffy,  2009).  This  research  addressed  the  question: 
Given  a  “real-world”  Army  institutional  training  environment,  is  there  an  operationally 
significant  interaction  between  type  of  instruction  (PCI  or  DI)  and  type  of  subject  domain  (well- 
defined  or  ill-defined)? 

Procedure: 

A  DI  based  and  a  PCI  based  version  of  an  existing  Army  training  module 
[Noncommissioned  Officer  (NCO)  Evaluation  Report  Preparation]  were  constructed  and  each 
administered  to  different  Infantry  Advanced  Leader  Course  classes.  A  common  post-test  addressing 
both  the  well-defined  and  the  ill-defined  elements  of  the  module  was  administered  to  all  students. 
Also,  both  the  DI  and  PCI  students’  attitudes  toward  the  effectiveness  of  the  module  were  assessed. 
The  hypotheses  were  that  (1)  both  DI  and  PCI  students’  performance  overall  would  benefit  from  the 
training,  (2)  DI  students  would  out-perform  PCI  students  on  tests  of  well-defined  elements,  while 
PCI  students  would  out-perform  DI  students  on  tests  of  ill-defined  elements  and  that  (3)  PCI 
students  would  have  a  more  positive  attitude  toward  PCI  than  the  DI  students  would  have  toward 
DI. 


In  parallel,  the  resources  required  to  construct  and  to  administer  and  maintain  the  existing 
NCO  Evaluation  Report  Preparation  module,  the  DI  version  of  the  module,  and  the  PCI  version  of 
the  module  were  surveyed  to  allow  a  rank  order  determination  of  the  resourcing  requirements  of  the 
module  versions. 

Findings: 

For  both  DI  and  PCI,  as  assessed  by  the  pre-test  and  post-test,  students  performed 
significantly  better  after  instruction,  with  neither  instructional  method  superior  to  the  other.  The 
expected  interaction  between  type  of  instruction  and  subject  domain  type  was  not  observed. 

Also,  PCI  students  consistently  valued  their  training  more  highly  than  did  DI  students,  although 
this  difference  was  marginally  significant. 
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Additionally,  the  PCI  module  was  found  to  require  the  most  resources,  the  DI  module, 
less,  and  the  existing  module  the  least  resources. 

The  overall  performance  improvement  regardless  of  instructional  method  indicates  that, 
as  hypothesized,  with  a  focused  development  effort,  it  is  possible  to  measurably  improve  the 
effectiveness  of  existing  training.  The  failure  to  find  an  interaction  between  instructional  method 
(DI  or  PCI)  and  subject  domain  type  (ill  or  well  defined)  should  be  interpreted  as  a  precaution: 
although  both  DI  and  PCI  methods  can  be  effective,  it  may  not  be  the  case  that  one  is  measurably 
more  effective  than  the  other,  even  when  training  subject  domain  type  is  taken  into  account. 

Considerations  for  trading  off  training  effectiveness  versus  required  training  resources 
were  addressed. 

Utilization  and  Dissemination  of  Findings: 

Results  were  briefed  to  the  Henry  Caro  Noncommissioned  Officer  Academy  (NCOA)  at 
the  United  States  (US)  Army  Maneuver  Center  of  Excellence,  Fort  Benning,  GA.  The  NCOA 
Advanced  Leaders  Course  has  incorporated  a  version  of  the  NCO  Evaluation  Report  preparation 
block  of  instruction  into  its  current  curriculum. 
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COMPARISON  OF  DIRECT  INSTRUCTION  AND  PROBLEM  CENTERED 
INSTRUCTION  FOR  ARMY  INSTITUTIONAL  TRAINING 

Introduction 

The  stated  purpose  of  the  updated  Army  Learning  Model  (ALM)  elaborated  in  The 
United  States  (U.S.)  Army  Learning  Concept  for  2015  (U.S.  Army,  201 1)  is  to  meet  the  “Army’s 
need  to  develop  adaptive,  thinking  Soldiers  and  leaders  capable  of  meeting  the  challenges  of 
operational  adaptability  in  an  era  of  persistent  conflict”  (p.  2).  In  fulfilling  this  purpose,  the 
ALM  prescribes  that  the  Anny  begin,  as  appropriate,  transforming  existing  instruction  and 
developing  new  “context-based,  collaborative,  problem-centered  instruction”  (p.  19). 

Problem-centered  instruction  (PCI)  is  an  instructional  approach  in  which  problem  solving 
is  used  as  a  vehicle  for  knowledge  acquisition  and  for  improvement  of  problem-solving  skills 
(Hmelo-Silver,  2004).  It  is  characterized  as  being  instructor  facilitated  (vice  instructor  led),  being 
focused  on  relevant  exemplar  problems,  and  emphasizing  the  learner’s  contextual  understanding 
of  the  subject  matter  (cf.  Duffy  &  Raymer,  2010).  As  such,  PCI  represents  a  sizable  departure 
from  the  Anny’s  historical  framework  of  direct  instruction  (DI),  which  in  contradistinction  has 
been  characterized  as  relatively  inflexible,  instructor  led,  and  focusing  on  individual  tasks, 
conditions,  and  standards  primarily  through  lecture  (U.S.  Army,  2011). 

Given  that  the  ALM  prescribes  the  use  of  PCI,  there  are  at  least  two  practical 
implementation  issues  the  Army  faces:  (1)  when  PCI  is  applied  appropriately,  what  is  the  gain  in 
training  effectiveness?  and  (2)  what  typical  cost  elements  might  be  associated  with  adding  PCI  to 
the  Army  institutional  training  environment? 

These  two  issues  are  inter-related.  The  first  issue  addresses  the  practicality  to  the  Army 
of  implementing  PCI.  Although  there  are  strong  voices  of  caution  (cf.  Kirschner,  Sweller  & 
Clark,  2006;  Mayer,  2004),  ALM  assumes  the  general  relative  superiority  of  problem-based 
instructional  approaches  compared  to  approaches  based  in  DI.  If  this  superiority  does  indeed 
hold  true,  then  to  objectively  justify  a  shift  to  PCI,  the  Army  should  be  able  to  show  that  with 
PCI  there  is  a  concomitant  gain  in  training  effectiveness. 

Moreover,  this  concomitant  gain  must  more  than  offset  the  resources  required  to  make 
and  to  maintain  the  shift  from  DI  to  PCI.  This  consideration  raises  the  second  issue  -  to  assess 
the  degree  of  resource  offset,  the  Army  needs  a  characterization  of  the  current  (Dl-related)  and 
proposed  (PCI-related)  elements  of  cost  associated  with  instruction.  Once  these  elements  are 
characterized,  Anny  costing  procedures  can  be  invoked  to  begin  detennining  actual  costs. 

Within  the  issue  of  assessing  the  effectiveness  of  PCI  and  DI  approaches  is  the  further 
issue  of  the  characteristics  of  the  material  to  be  learned,  or  the  learning  domain.  Some  domains, 
such  as  a  data  entry  task,  are  relatively  well  defined,  with  explicit  processes  and  standards. 

Other  domains,  such  as  crafting  written  descriptive  evaluations  of  Soldiers’  job  perfonnance,  are 
relatively  ill  defined,  with  few  explicit  processes  or  standards.  In  general,  DI  is  assumed  more 
appropriate  for  a  well-defined  domain  of  learning,  while  PCI  is  assumed  more  appropriate  for  an 
ill-defined  domain  of  learning  (Clark,  2004;  Tobias  &  Duffy,  2009).  Thus,  current  theoretical 


1 


views  of  PCI  and  DI  would  dictate  that  any  comparison  between  the  two  must  take  into  account 
their  respective  assumed  strengths  and  weaknesses  relative  to  the  degree  of  “definition”  of  the 
learning  domain(s)  used  in  the  comparison. 

This  investigation  used  an  existing  Anny  block  of  instruction,  with  some  Soldiers 
instructed  via  DI  methods  and  other  Soldiers  via  PCI  methods.  Then,  all  Soldiers’  post¬ 
instruction  performance  on  all  the  well-defined  and  on  all  the  ill-defined  elements  of  the 
instructional  domain  was  assessed.  DI  Soldiers’  performance  on  ill-defined  vs.  well  defined 
elements  was  compared,  as  was  PCI  Soldiers’  performance  on  ill-defined  vs.  well  defined 
elements.  Additionally,  during  preparation  of  the  instructional  materials  and  during  actual 
presentation  and  assessment  of  instruction,  the  resources  required  for  each  method  were 
characterized  and  roughly  estimated. 

The  following  sections  give  in  more  detail  the  background  of  the  research,  the  research 
methods  used,  the  research  results,  and  the  implications  of  the  results. 


Background 

Considerations  in  Applying  the  2015  Learning  Model 

The  application  of  the  2015  learning  model  in  the  Anny  institutional  training 
environment,  while  theoretically  appealing,  will  be  challenging  to  execute  given  the  existing 
learning  culture  and  practical  constraints  that  impact  and  shape  classroom  training.  While  the 
problem-centered  approach  (and  its  many  variants)  promoted  by  the  2015  learning  model  can  be 
effective  in  improving  adaptive  responding  it  is  also  clear  that  this  approach  may  not  be  best 
suited  for  training  all  skill  areas,  e.g.,  procedural  based  tasks  with  clear  solutions,  or  all  students, 
e.g.,  advanced  learners  (Clark,  2004). 

Recent  research  (e.g.,  Pleban,  Graves  et  ah,  2011;  Pleban,  Vaughn,  Sidman,  Semmens  & 
Geyer,  201 1;  &  Tucker,  Semmens,  Sidman,  Geyer,  &  Vaughn,  2011;  U.S.  Army,  2011) 
provides  some  support  for  Clark’s  (2004)  major  conclusions,  indicating  that  the 
effectiveness/utility  of  a  specific  training  strategy  (i.e.,  direct  instruction  versus  problem 
centered)  will  depend  on  a  number  of  factors  such  as  learning  objectives,  time  allotted  for 
instruction,  available  resources,  preparation  of  instructor,  and  level  of  experience  of  the  learner. 

Framework  for  Transforming  Army  Training  and  Education  and  Development  (ATED) 

Previous  Army  publications  have  emphasized  the  importance  of  multiple  instructional 
strategies  and  selecting  the  best  approach  based  on  such  factors  as  course  content,  training 
objectives,  and  learner  experience.  As  a  prime  example,  Training  and  Doctrine  Command 
(TRADOC)  Pamphlet  350-70-4,  Guide  to  Army  Training  and  Education  Development:  Process, 
Frameworks,  Models,  and  Efficiencies  (U.S.  Anny,  2009)  identifies  DI  and  PCI  as  the  major 
models  for  instruction.  These  two  instructional  approaches  are  briefly  described  in  the  following 
sections. 
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Direct  instruction-DI:  Overview.  Guided  by  Merrill’s  (2009)  first  principles  of 
learning,  DI  is  often  associated  with  “modeling”  expert  perfonnance.  This  approach  is  based  on 
the  premise  that  someone  can  learn  to  perform  a  task  or  solve  a  problem  in  the  same  way  as  an 
expert.  DI  represents  a  part  to  whole  approach  to  learning.  It  involves  dividing  the  learning 
material  into  chunks,  connecting  new  knowledge  to  existing  knowledge  (in  the  learner),  typically 
through  lecture,  and  demonstrating  how  to  perform  each  chunk.  This  is  followed  by  practice 
opportunities.  Critical  infonnation  can  be  provided  through  varied  platforms  to  include  not  only 
lecture  but  also  readings  (e.g.  case-based  studies),  handouts,  and  interactive  multimedia 
instruction. 

Feedback  is  provided  during  practice  and  learners  are  assessed  on  their  ability  to  perfonn 
the  task  or  solve  the  problem.  After  practicing  small  chunks,  learners  are  presented  with  another 
problem  scenario  and  allowed  to  practice  solving  the  entire  problem. 

It  should  be  noted  that  DI  is  sometimes  regarded  as  the  “lecture”  approach  in  which  the 
instructor  provides  knowledge  to  a  passive  learner.  However,  good  DI  involves  more  than  just  a 
lecture.  It  requires  well  trained,  knowledgeable  instructors  who  are  actively  involved  in 
demonstrations,  monitoring  practice,  evaluating  learning,  and  providing  the  appropriate  feedback 
(i.e.,  infonnation  that  helps  learners  identify  the  strengths  and  weaknesses  in  their  performance 
while  providing  strategies  for  improvement). 

Guidelines  for  when  to  use  DI  as  the  instructional  strategy.  TRADOC  Pamphlet  350- 
70-4  (2011)  provides  clear  guidelines  for  when  to  select  DI  as  the  instructional  strategy.  First, 
DI  should  be  considered  when  the  objective  is  to  prepare  students  to  perfonn  a  task  or  solve  a 
problem  in  the  same  manner  as  someone  who  is  an  expert  in  that  content  area.  Second,  DI  is 
most  often  the  more  appropriate  choice  when  the  training  objective  involves  learning  to  solve 
well-structured  problems  (e.g.,  learning  how  to  operate  a  lathe,  repair  a  radio)  with  a  clear 
solution.  Finally,  DI  is  more  appropriate  for  students  who  are  novices  in  the  learning  topic. 

Cautions  when  using  DI  as  an  instructional  strategy.  DI  may  be  inappropriate  if  the 
target  audience  is  already  expert  in  the  subject  being  taught.  Presentation  of  basic  factual 
information,  concepts  and  simplistic  (part)  tasks  risk  losing  the  attention  of  the  experienced 
audience  who  may  view  the  instruction  as  providing  little  challenge  or  incentive  to  excel.  In 
addition,  instructors  must  be  prepared  to  execute  DI  using  instructional  interventions  that  engage 
learners  in  all  aspects  of  the  learning  event.  Learning  how  to  conduct  good  DI  must  be  clearly 
addressed  in  faculty  training/education  programs. 

Problem-centered  instruction-PCI:  Overview.  PCI  represents  a  whole  to  part 
approach  to  learning  in  that  the  students  are  presented  with  a  problem  or  several  problems  first 
which  requires  them  to  develop  a  solution(s)  and/or  identify  underlying  principles  for  addressing 
similar  types  of  problems  in  future,  but  differing  situations.  The  instructor’s  role  in  PCI  is  to 
serve  as  a  facilitator  of  knowledge  as  opposed  to  directing  the  students’  thinking.  The  instructor 
provides  lectures  on  key  learning  points  after  the  students  have  had  an  opportunity  to  work  on 
the  problem(s)  and  to  formulate  a  solution. 
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Guidelines  for  when  to  use  PCI  as  the  instructional  strategy.  PCI  should  be  considered 
when  the  objective  is  to  prepare  students  to  function  in  uncertain,  ambiguous  or  evolving 
situations,  e.g.,  leadership,  stability  operations,  organizational  effectiveness.  In  general,  PCI  is 
viewed  as  most  appropriate  when  addressing  ill-structured  problems,  e.g.,  leadership,  where 
there  is  no  clear  or  single  correct  solution. 

Cautions  when  using  PCI  as  an  instructional  strategy.  PCI  is  more  appropriate  when 
the  students  are  already  expert  (or  are  familiar  with  the  area).  The  rationale  provided  is  that  the 
more  sophisticated  learners  will  view  the  requirements  to  investigate,  research  and  develop 
solutions  for  problems  not  seen  before  as  challenges  to  grow  and  excel  within  their  field.  They 
will  thus  be  less  likely  to  be  overwhelmed  (and  more  likely  to  leam)  than  students  unfamiliar 
with  the  topic  area  in  question  (U.S.  Department  of  the  Army,  2009). 

PCI  instructors.  As  was  the  case  for  DI,  to  conduct  an  effective  course  based  on  PCI 
principles  will  require  extensive  instructor  preparation.  Cianciolo,  Grover,  Bickley,  and 
Manning  (2011)  identified  key  instructor  competencies  that  are  needed  to  orchestrate  an  effective 
PCI  driven  course.  These  competencies  fall  into  two  categories:  classroom  management  and 
learning  facilitation.  With  regard  to  classroom  management  activities,  developing  productive 
PCI-based  learning  requires  that  the  instructor  prioritize  learning  objectives,  structure  problems 
and  assessment  activities,  and  anticipate  the  most  likely  areas  of  student  difficulty.  Specific 
competencies  associated  with  classroom  management  include:  specifying  the  knowledge  and 
skill  acquisition  objectives  for  the  course;  developing  problems  to  trigger  and  situate  the  learning 
process;  and  constructing  a  problem  “roadmap”  that  provides  structure  that  guides  the 
assessment  and  understanding  of  students  and  scaffolds  learners’  development  of  cognitive  skill. 

The  second  competency  category,  learning  facilitation,  refers  to  what  instructors  do  in 
real  time  to  insure  that  students  remain  active  and  in  charge  of  their  learning  while  keeping 
learning  on  track  to  meet  course  requirements.  Key  instructor  competencies  for  this  category 
include:  real-time  monitoring  and  assessment  of  student  perfonnance;  role  modeling  the 
problem-solving  process  to  help  students  understand  how  to  approach  and  solve  complex 
problems;  and  facilitating  group  discussion  and  collaboration  in  order  to  support  knowledge 
acquisition  and  give  students  a  model  of  successful  collective  problem  solving. 

Training  Issues  and  Implications 

Careful  review  of  TRADOC  Pam  350-70-4  (U.S.  Anny,  2009)  suggests  that  the  PCI 
approach  described  comprises  a  broad  range  of  student-centered  instructional  methods.  PCI  is 
based  in  research  which  suggests  that,  by  having  them  learn  through  problem-solving 
experiences,  students  can  leam  both  content  as  well  as  thinking  strategies.  To  expand  on  the 
description  provided  earlier,  in  PCI,  students  leam  through  facilitated  problem  solving.  More 
specifically,  learning  centers  on  a  complex  problem  that  does  not  have  a  single  correct  answer. 
Students  work  in  collaborative  groups  to  identify  what  they  need  to  leam  to  solve  a  problem. 
They  engage  in  self-directed  learning  and  then  apply  their  new  knowledge  to  the  problem.  They 
then  reflect  on  what  they  learned  and  the  effectiveness  of  the  strategies  employed.  In  this 
approach,  the  instructor’s  role  is  to  facilitate  the  learning  process  rather  than  provide  knowledge. 
Because  students  are  self-directed,  managing  their  learning  goals  and  strategies  to  solve  ill 
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defined  problems,  they  are  able  to,  presumably,  acquire  the  skills  needed  for  lifelong  learning 
(Hmelo-Silver,  2004;  see  also  Hmelo-Silver,  Duncan,  &  Chinn,  2007). 

Currently,  there  is  little  empirical  research  performed  on  the  training  value/impact  of 
PCI  when  implemented  in  an  Army  institutional  classroom  setting.  Additionally,  issues 
involving  classroom  organization  such  as  fixed  time  length  instructional  periods  in  military 
classroom  training  environments  (of  shorter  duration  than  might  be  optimal  for  PCI),  skill  levels 
of  current  instructors  to  serve  as  course  facilitators  versus  lecturers,  and  the  relatively  high 
student/instructor  ratio  (40: 1)  in  many  military  classrooms  may  limit  the  applicability  of  this 
particular  approach.  These  practical  limitations  have  led  some  researchers  (Tucker  et  ah,  2011; 
Pleban,  Graves  et  ah,  2011;  Pleban,  Vaughn  et  ah,  2011)  to  examine  the  feasibility  of 
implementing  alternative  PCI  strategies  in  Anny  institutional  classroom  settings. 

Contrasting  cases/invention:  Overview.  One  strategy  that  has  received  scrutiny  as  one 
of  the  approaches  to  PCI  is  contrasting  cases/invention.  Contrasting  cases/invention  are  two 
instructional  design  features  used  to  enhance  deep  understanding  of  subject  matter  materials. 

The  approach  was  developed  to  help  people  construct  new  knowledge  for  themselves  and 
become  more  adaptive/effective  problem  solvers  (Schwartz  &  Bransford,  1998;  Schwartz  & 
Martin,  2004).  A  key  objective  of  this  problem-centered  approach  is  to  optimize  the  use  of 
lectures/reading  materials  to  develop  these  skills.  Schwartz  and  Bransford  argue  that  the  value 
of  lectures  can  be  enhanced  if  the  student  is  able  to  map  information  from  the  lecture  or  text  into 
the  knowledge  of  the  problem  situation  that  they  have  already  developed  as  a  result  of  their  prior 
experiences.  A  key  assumption  of  this  strategy  is  that  the  student  can  activate  the  prior 
knowledge.  Schwartz  and  Bransford  propose  a  way  for  activating  this  prior  knowledge  through 
the  use  of  contrasting  cases/invention.  Based  on  theories  of  perceptual  learning  that  emphasize 
differentiation  (e.g.,  Bransford,  Franks,  Vye,  &  Sherwood,  1989),  providing  students  with 
opportunities  to  analyze  sets  of  contrasting  cases  (e.g.,  analyzing  the  results  from  different 
experiments,  key  aspects  of  different  theoretical  models)  can  help  them  become  sensitive  to 
information  that  they  might  not  otherwise  notice.  Contrasting  cases  help  attune  people  to 
specific  features  and  dimensions  that  make  the  cases  distinctive.  The  refined  infonnation 
provides  the  foundation  for  guiding  other  activities  such  as  creating  images,  elaborating,  and 
generating  questions,  which  can  enhance  development  of  adaptive  problem  solving  skills. 

According  to  Schwartz  and  Martin  (2004),  contrasting  cases  can  help  people  pick  up  or 
notice  distinctive  features;  however,  it  is  their  actions  that  are  critical  for  helping  them  discern 
the  structures  that  organize  those  features.  To  make  contrasting  cases  effective,  learners  need  to 
undertake  productive  activities  that  lead  them  to  notice  and  account  for  contrasts  in  the  different 
cases.  Schwartz  and  Martin  use  the  term  invention  to  describe  this  process.  Invention  involves 
production  activities,  like  inventing  solutions  that  can  be  particularly  beneficial  for  developing 
early  knowledge  and  facilitating  learning.  These  solutions  could,  for  example,  be  in  the  fonn  of 
graphs,  or  general  formulas.  Invention  can  help  develop  and/or  clarify  interpretations  of  the 
problem  in  question  by  forcing  students  to  notice  inconsistencies  in  their  approach  or  mental 
model  of  their  solution  and  work  to  reconcile  them.  This,  in  turn,  provides  the  knowledge  that 
will  prepare  them  to  learn  from  subsequent  instruction  (lectures)  with  deeper  understanding 
(Schwartz,  Sears,  &  Chang,  2008). 
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To  optimize  deep  understanding  of  the  subject  matter  material,  Schwartz  and  colleagues 
advocate  a  particular  sequencing  of  events.  Students  first  try  to  solve  novel  problems  without 
guidance/instruction,  to  “invent”  potential  solutions  to  the  problem.  Then,  they  receive  DI  and 
demonstrations  regarding  the  tasks.  Finally,  they  apply  what  they  have  learned  to  novel 
situations.  For  example,  students  might  analyze  data  sets  from  classical  experiments  and  attempt 
to  graphically  display  the  general  phenomena  from  the  data.  Or,  they  might  be  asked  to  invent  a 
model  or  formula  that  will  accurately  describe  the  concept  (e.g.,  reliability  or  correlation).  This 
would  be  followed  by  a  lecture  and  (sometimes)  class  discussion  and  possibly  a  demonstration. 
Finally,  students  would  be  presented  with  new  problems  and  asked  to  make  predictions 
concerning  the  outcomes  of  new  experiments  or  apply  the  fonnula  or  model  to  solve  another 
(novel)  problem  (Schwartz,  Bransford,  &  Sears,  2005;  Schwartz  &  Martin,  2004). 

While  contrasting  cases/invention  is  a  critical  part  of  Schwartz’  approach,  the  lecture 
component  (DI)  is  equally  valuable.  It  offers  a  higher  level  explanation  of  the 
concept/phenomena  that  would  be  quite  difficult  and  time  consuming  for  the  student  to  discover 
on  his  or  her  own.  The  higher  level  explanation  is  important  because  it  provides  a  generative 
framework  that  can  extend  one’s  understanding  beyond  the  specific  cases  that  have  been 
analyzed  and  experienced  (Schwartz  &  Black,  1996)  and  thus,  enhances  adaptive  problem 
solving  (transfer).  By  sequencing  the  lecture  following  invention/contrasting  cases,  a  “time  for 
telling”  is  created  that  increases  the  learning  value  of  the  lecture  as  students  are  now  better 
prepared  to  grasp  the  deeper  implications  of  the  lecture  as  a  result  of  their  earlier  discovery 
activities  (Schwartz  &  Bransford,  1998).  Schwartz,  Bransford,  and  Sears  (2005)  present 
evidence  that  the  most  effective  design  combination  includes  both  opportunities  for  invention 
and  analysis  (contrasting  cases)  followed  by  opportunities  for  learning  efficient  solutions  derived 
by  experts  (typically)  presented  in  lecture  format. 

To  date,  contrasting  cases/invention  has  been  used  to  train  ethical  decision-making  skills 
with  United  States  Military  Academy  cadets  (Pleban,  Graves  et  ah,  2011),  and  mission  planning 
skills  with  second  lieutenants  attending  the  Infantry  Basic  Officer  Leader  Course  (Tucker  et  ah, 
2011;  Pleban,  Vaughn  et  al.,  2011)  with  mixed  success.  Tucker  et  al.  found  this  approach 
yielded  the  highest  self-report  of  training  effectiveness  and  was  more  engaging  for  students. 
Students  also  rated  this  approach  highest  with  regard  to  adequacy  of  coverage  of  course  topics. 
Pleban,  Vaughn  et  al.  found  this  approach  to  be  ineffective  in  training  mission  planning  skills.  A 
weakness  of  the  invention/contrasting  cases  framework  was  the  high  cognitive  loads  placed  on 
the  students.  Also,  the  brevity  of  the  training  allotted  in  these  investigations  (one  day  for  Pleban, 
Vaughn  et  al.  and  two  days  for  Tucker  et  al.)  may  have  played  a  major  role  in  explaining  the  lack 
of  improvement  from  the  students  on  the  objective  performance  measures. 

Pleban,  Graves  et  al.  (2011)  found  the  contrasting  cases/invention  approach  to  be 
effective  in  training  ethical  decision-making  skills  but  the  training  involved  repeated  cycles  of 
invention/contrasting  cases  over  a  period  of  sixteen  weeks.  Although  total  training  duration  may 
be  problematical,  there  are  a  few  advantages  of  this  approach  for  use  in  a  military  classroom 
training  environment.  The  module  design  can  readily  fit  established/conventional  (50-60 
minutes)  time  blocks  and  not  negatively  impact  course  throughput.  Another  advantage  of  the 
invention-contrasting  cases  approach  is  of  its  partial  reliance  on  lectures.  Military  course 
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instructors  would  be  more  comfortable  with  this  approach  versus  serving  exclusively  as  a  PCI 
facilitator. 

Framework  for  Costing  Instructional  Methodologies 

As  indicated  above,  the  decision  to  change  from  one  instructional  methodology  to 
another  is  based  not  only  on  the  relative  training  effectiveness  of  the  methodologies,  but  also  on 
the  resources  required  (1)  to  change  from  one  to  another  and  (2)  to  subsequently  maintain  the 
replacement  methodology.  The  universe  of  training  resources  is  large,  and  includes  both 
tangible  items,  such  as  troop  transport  vehicles,  and  intangible  items,  such  as  instructor  personal 
characteristics. 

Given  the  number  and  diversity  of  resources  that  must  be  considered  for  costing  Anny 
training,  there  is  a  need  to  adopt  a  comprehensive  model  or  framework  that  can  be  used  to 
methodically  suggest  and  categorize  candidate  methodologies.  The  framework  should  be  such 
that  it  accounts  for  impact  on  resources  within  the  instructional  course,  on  resources  that  directly 
support  the  course,  and  on  resources  that  indirectly  support  the  course.  It  also  should  take  into 
account  the  impact  of  the  methodology  on  the  student  as  well  as  the  overall  training  system. 

The  framework  selected  was  a  modification  of  the  Federal  Enterprise  Architecture 
Performance  Reference  Model  or  PRM  (U.S.  Office  of  Management  and  Budget,  2007).  PRM  is 
designed  to  identify  relationships  between  resource  inputs,  outputs  and  desired  outcomes.  A 
modified,  high  level  depiction  of  the  framework  is  provided  in  Figure  1. 


Figure  1.  Performance  Reference  Model  (U.S.  Office  of  Management  and  Budget,  2007). 
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This  effort  used  the  PRM  framework  as  a  guide  to  ensure  all  of  the  stakeholder  areas 
were  considered.  The  primary  stakeholders  in  this  case  were  not  just  the  Noncommissioned 
Officer  Academy  (NCOA)  and  Noncommissioned  Officer  (NCO)  students,  but  also  the  units 
who  will  receive  the  NCOA  graduates  and  the  Soldiers  that  the  NCOs  will  lead  and  evaluate. 
Terminology  from  the  PRM  was  adjusted  to  accommodate  Anny  tenninology  as  was  the 
process,  with  the  recognition  that  Anny  training  is  not  conducted  in  accordance  with  a  standard 
business  model  (e.g.,  in  most  cases,  the  primary  customer  of  Anny  training  is  required  to 
participate  and  resources  are  constrained  by  levels  well-above  the  actual  implemented). 

In  the  remainder  of  this  report,  the  PRM  inputs  are  treated  in  the  Methods  section,  the 
outputs  are  treated  in  the  Results  section,  and  the  outcomes  are  treated  in  the  Results  and 
Discussion  sections. 

Research  Objectives 

The  objectives  of  the  present  research  were  to  expand  on  earlier  research  by  examining 
the  training  value/impact  of  the  contrasting  cases/invention  strategy  in  a  typical  Anny 
institutional  classroom  setting  as  well  as  investigating  the  differential  effectiveness  of  DI  and 
PCI  approaches  for  training  different  types  of  tasks,  i.e.,  procedural/factual  tasks  with  a  well- 
defined  correct  solution  versus  ill-structured  tasks  with  no  clear  cut  solution. 

Given  the  research  was  also  motivated  by  considerations  of  implementing  ALM,  there 
were  two  additional  objectives.  The  first  of  these  was  to  situate  the  work  in  a  venue  realistic  to 
Anny  institutional  training.  That  is,  an  effort  was  made  to  identify  an  existing  course  of 
instruction  and  to  incorporate  an  existing  block  of  training  into  the  research.  And,  within  that 
course  of  instruction,  students  normally  enrolled  in  the  course  were  to  be  used  as  experimental 
subjects.  An  interaction  was  predicted  with  students  receiving  a  DI  module  perfonning  better  on 
tasks  requiring  the  recall  of  facts/declarative  knowledge  than  those  receiving  a  prototype  PCI 
module  and  with  students  receiving  the  PCI  module  perfonning  better  on  the  more  ill-structured 
tasks  than  students  receiving  the  DI  module. 

The  second  of  these  objectives  was  to  begin  to  identify  and  characterize  the  institutional 
cost  factors  that  should  be  taken  into  account  in  making  decisions  to  implement  PCI  for  existing 
training.  Given  that  a  PCI  approach  for  specific  training  may  result  in  more  effective  or  efficient 
training,  before  a  decision  is  made  to  convert  to  PCI,  there  must  be  evidence  that  the  gains  due  to 
PCI  are  not  negated  by  a  concomitant  increase  in  cost  to  train. 


Method 


Training  Effectiveness 


Design  Overview.  The  course  of  instruction  selected  was  Military  Occupational 
Specialty  (MOS)  1  IB  Phase  2  Advanced  Leaders  Course  (ALC).  1  IB  ALC  is  conducted  by  the 
Henry  Caro  NCO  Academy  (NCOA)  at  the  Maneuver  Center  of  Excellence,  Fort  Benning,  GA. 
The  instructional  cadre  are  1  IB  Staff  Sergeants  (SSGs)  and  Sergeants  First  Class  (SFCs).  1  IB 


ALC  students  are  mostly  SSGs,  along  with  some  senior  Sergeants  (SGTs).  1  IB  ALC  course  of 
instruction  consists  of  multiple  4-hour  blocks  of  instruction  in  topics  that  are  critical  to  an  1  IB 
SSG’s  duties.  A  block  of  instruction  may  be  delivered  by  en  masse  lecture  to  up  to  160  students 
or,  in  most  cases,  to  small  16-person  groups. 

The  topic  selected  was  preparation  of  noncommissioned  officer  evaluation  reports 
(NCOER).  The  NCOER  is  an  annual  evaluation  addressing  the  NCO’s  perfonnance  and 
suitability  for  increase  in  responsibilities  and  in  rank.  In  a  typical  unit,  subordinates’  NCOERs 
are  prepared  by  SSGs  (raters)  based  on  the  raters’  assessment  of  their  subordinates  during  the 
period  being  reported.  The  two  fonns  associated  with  NCOER  preparation,  the  “NCOER 
Counseling  and  Support  Form”  and  “NCO  Evaluation  Report,”  are  shown  in  Appendix  A. 

While  there  are  many  well-defined,  procedural  components  involved  in  writing  a  good 
NCOER,  there  are  additional  ill-defined  requirements  involved  as  well  in  writing  narrative 
“bullet”  statements  to  support  the  ratings  on  various  NCO  tasks,  leadership  dimensions  and 
Anny  values.  Thus,  the  topic  of  NCOER  preparation  was  selected  based  on  its  being  composed 
of  both  well-defined  and  ill-defined  tasks. 

The  1  IB  ALC’s  then  current  version  of  NCOER  preparation  instruction  was  conducted  in 
a  DI  4-hour  mass  lecture  format,  with  upwards  of  160  students  in  the  class.  The  lecture 
comprised  minimal  student  active  participation  and  neither  fonnative  nor  summative  assessment. 
In  the  research  team’s  opinion,  this  was  a  relatively  inefficient  application  of  DI,  and  a 
comparison  of  the  effectiveness  of  the  lecture  version  with  a  PCI  version  would  likely  trivially 
and  uninfonnatively  favor  PCI.  Therefore,  as  part  of  the  design,  an  experimental  DI  version  of 
the  block  of  instruction  was  constructed,  drawing  on  the  pre-ALM  Army  guidelines  for 
developing  training  (U.S.  Anny,  1999).  The  design  consideration  was  that  the  PCI  and  the  DI 
instances  should  be  based  on  the  best  practices  for  both  frameworks.  Content  and  process  of  the 
two  experimental  instances  are  outlined  below. 

A  mixed  factor  design  was  employed,  with  DI  and  PCI  experimental  conditions  and  a 
control  condition.  The  experimental  conditions  comprised  six  1  IB  ALC  small  groups 
(approximately  sixteen  students  per  small  group)  randomly  assigned  to  the  experimental 
conditions,  two  groups  to  receive  the  DI,  two  the  PCI  training  module,  and  two  the  then  current 
lecture  module.  Students  in  both  the  DI  and  PCI  training  conditions  completed  an  NCOER 
pretest,  participated  in  the  instruction,  and  then  completed  an  NCOER  posttest.  Students  in  the 
control  condition  received  the  NCOER  posttest  only. 

Experimental  participants.  Participants  in  the  DI  and  PCI  conditions  were  67  NCOs 
attending  1  IB  ALC. 

Tables  1-3  summarize  demographic  and  educational  information  of  the  ALC  students 
who  took  part  in  this  research.  The  items  in  Table  3  refer  to  items  on  the  “NCO  Evaluation 
Report”  fonn  in  Appendix  A.  The  data  of  several  participants  were  screened  out  for  being 
incomplete  and/or  otherwise  indicating  careless  responding  (i.e.,  invariant  data,  respondents 
circling  entire  columns  of  responses,  etc.;  see  Dollinger  &  DiLalla,  1996).  After  this  screening 
process,  the  sample  consisted  of  61  participants. 
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Table  1.  Summary  Demographic  Information  for  the  Infantry  ALC Students 


SGT  (n  = 

18) 

SSG  (n 

=  40) 

Mean  Time  in  Service  (months) 

75 

94 

Mean  Time  in  Grade  (months) 

28 

20 

Civilian  Education: 

GED 

1 

6% 

4 

10% 

HS  Diploma 

10 

56% 

13 

33  % 

Some  College 

7 

39% 

18 

45  % 

Associate  or  Bachelor  Degree 

0 

0% 

5 

13  % 

Table  2.  Relevant  NCOER  Education/Experience  Reported  by  In  fantry  ALC  Students 
NCOER  Relevant  Education/Experience  (%) _ 


Received  Recognition  for  Writing  1 5 

Taken  College  Level  Writing  Course  25 

Regularly  Conduct  Quarterly  Counseling  83 

Regularly  Use  the  NCOER  Support  Form  43 

Note',  n  =  60 


Table  3.  Relevant  NCOER  Experience  Reported  by  In  fantry  ALC  Students:  I 


NCOER  Relevant  Experience  (%) 

Me 

Platoon 

Sgt 

Co 

Cmdr 

Above/ 

Unk 

Who  Completed  the  Administrative  Sections 

63 

33 

0 

4 

(Parts  I  &  II)? 

Who  Developed  Bullet  Comments  (Part  IV)? 

71 

25 

0 

4 

Who  Detennined  the  Ratings? 

44 

43 

6 

7 

Note :  n  =  54 
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Table  4.  Relevant  NCOER  Experience  Reported  by  Infantry  ALC  Students:  II 


NCOER  Relevant  Experience  (%) 

1 

2-3 

4-5 

5+ 

How  many  NCOERs  have  you  been  a  rater  for?  (n  =  45) 

35.6 

42.2 

11.1 

11.1 

How  many  NCOERs  have  you  received?  (n  =  59) 

10.2 

59.3 

16.9 

13.6 

Control  participants.  The  control  group  consisted  of  30  ALC  students.  All  had 
previously,  as  part  of  the  ALC  POI,  received  the  then-current  4-hour  block  of  instruction 
covering  principles  of  Army  writing  and  preparation  of  NCOERs. 

Instructors.  The  same  instructor  conducted  both  the  DI  and  the  PCI  classes.  A  member 
of  the  research  team,  this  instructor  was  a  trained,  qualified  educator,  with  experience  in  both  DI 
and  PCI  methods.  The  DI/PCI  instructor  is  also  considered  a  subject  matter  expert  (SME)  in  the 
area  of  NCOER  preparation.  A  different  SME  instructor  not  associated  with  the  research  team 
conducted  training  for  the  control  group  as  part  of  the  normal  1  IB  ALC  cadre. 

Instruments.  The  experimental  groups  were  administered  a  demographic  questionnaire 
and  pretest  prior  to  instruction,  and  a  posttest  and  post  training  questionnaire  subsequent  to 
instruction.  The  control  group  was  administered  the  posttest  only. 

Demographic  Questionnaire.  The  Demographic  Questionnaire  consisted  of  1 1  items 
presented  in  multiple  choice  or  short  answer  formats.  The  first  10  items  addressed  students’ 
general  background  and  their  experience  in  preparing  NCOERs.  The  eleventh  item  was  a  one- 
sentence  writing  sample.  The  entire  questionnaire  is  at  Appendix  B. 

Pretest.  The  Pretest  consisted  of  1 7  items  presented  in  multiple-choice  or  short-answer 
formats.  Pretest  (and  posttest)  content  was  validated  by  a  panel  of  subject  matter  experts 
consisting  of  a  senior  active  duty  trainer,  a  retired  Sergeant  First  Class  (SFC)  and  two  retired 
First  Sergeants  (1SG),  all  with  broad  experience  in  operational  unit  leadership  positions.  The 
context  of  the  pretest  involved  having  the  students  role  play  a  squad  leader  faced  with 
completing  an  NCOER  for  a  fictitious  Soldier  in  their  squad.  Students  were  provided  with  a 
partially  completed  NCO  Evaluation  Report  (DA  Fonn  2166-8),  a  partially  completed  NCOER 
Counseling  and  Support  Fonn  (DA  Fonn  2166-8-1),  three  Developmental  Counseling  Forms 
(DA  Form  4856)  and  the  fictitious  Soldier’s  file  containing  miscellaneous  orders,  records,  and 
notes.  In  addition,  they  were  provided  with  supplementary  NCOER  reference  material  (AR  623- 
3  and  DA  Pam  623-3  to  help  them  answer  selected  questions  on  the  pretest.)  The  Pretest  (shown 
in  Appendix  C)  consisted  of  three  sections: 

•  Rules  and  Principles  for  Writing  NCOER  Bullets  (6  items) 

•  NCOER  Knowledge  and  Bullet  Writing  (8  items  based  on  the  fictitious  Soldier) 

•  Rules  and  Principles  of  the  NCOER  Evaluation  Reporting  System  (3  items) 
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Posttest.  The  Posttest  consisted  of  1 5  items  presented  in  multiple  choice  or  short  answer 
formats.  The  Posttest  was  structured  in  a  similar  fashion  to  the  Pretest  but  the  scenario  based 
materials  involved  a  different  (fictitious)  Soldier.  See  Appendix  D  for  the  items. 

Post  Training  Questionnaire.  The  Post  Training  Questionnaire  consisted  of  21  items 
presented  in  Likert-type  or  short  answer  fonnat.  The  questionnaire  consisted  of  two  sections 
addressing  Soldiers’  perceived  level  of  understanding  of  the  NCOER  process  before  and  after 
instruction  and  their  attitudes  toward  the  effectiveness  of  the  instruction.  See  Appendix  E  for  the 
items. 


Procedure.  The  overall  sequence  of  events  for  addressing  training  effectiveness  is  given 
in  Figure  2  below.  Details  of  the  figure  appear  above  and  in  the  sections  following  the  figure. 


i - 1 

S  Demographic  j 
!  Questionnaire  ! 

■  _ i 

i - 1 

j  Demographic  | 
!  Questionnaire  ! 

■  _ i 


Pretest 


Pretest 


Dl  Instruction 


PCI  Instruction 


Posttest 


Posttest 


o 

CuO  ^ 
C  ro 

£  i 

TO  o 
lT  t* 

o  3 
q.  O 


j  Army  Writing  j 
land NCOER  j 

j  Prep  (4  hrs)  ! 

Figure  2.  Design  of  training  effectiveness  assessment. 

Experimental  condition:  PCI.  Students  arrived  at  one  of  the  ALC  small  group 
classrooms  as  one  group  (approximately  16  students  per  group).  The  students  were  first  briefed 
on  the  purpose  of  the  experiment  (i.e.,  to  prepare  the  rater’s  portion  of  an  NCOER  with  a  specific 
focus  on  writing  bullets  and  making  accurate  ratings).  They  then  completed  the  pretest  and  the 
Demographic  Questionnaire. 

Next,  the  class  was  divided  into  two-man  teams.  As  part  of  an  instructional  scenario  that 
they  were  taking  over  the  position  of  a  fictional  squad  leader  who  had  just  been  transferred,  each 
team  was  provided  with  forms/information  prepared  by  the  former  squad  leader  for  two  fictitious 
Soldiers  ,  i.e.,  NCO  Evaluation  Reports  (DA  Form  2166-8),  partially  completed  NCOER 
Counseling  and  Support  Forms  (DA  Form  2166-8-1),  initial  counseling,  Developmental 
Counseling  Forms  (DA  Form  4856)  and  the  Soldiers’  files  with  miscellaneous  orders,  records, 
and  notes.  The  student  teams  read  through  the  folders  and  contrasted  profiles  of  the  two 
fictitious  Soldiers  to  help  gauge  their  understanding  of  the  Soldiers’  performance  and  to  assess 


Posttest 
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the  former  squad  leader’s  ability  to  write  good  bullets  and  provide  accurate  assessments, 
counseling  and  follow-up  feedback. 

The  instructor  then  facilitated  a  group  discussion  on  the  evaluations  of  the  two  fictitious 
Soldiers  and  the  fonner  squad  leader’s  efforts  at  preparing  NCOERs.  Following  the  discussion, 
the  teams  then  wrote  bullets  on  specific  values,  attributes,  skills,  and  actions  (on  acetate  sheets) 
listed  in  Parts  IV  and  V  of  the  NCO  Evaluation  Report.  Each  team  addressed  a  different 
category,  e.g.,  Anny  values  -  honor,  integrity;  NCO  responsibilities  -  competence,  physical 
fitness,  leadership,  training  responsibility  and  overall  perfonnance  and  potential.  Bullet  writing 
addressed  the  invention  phase  of  the  PCI  module. 

After  each  team  had  written  its  bullet  for  an  assigned  value  or  NCO  responsibility,  the 
instructor  collected  each  acetate  sheet  from  the  teams  and  displayed  them  (separately)  on  an 
overhead  projector.  The  class  reviewed  each  bullet  and  provided  feedback.  The  instructor 
guided  the  discussion.  After  all  bullets  had  been  presented,  the  instructor  then  provided  a  brief 
review  (lecture)  to  reinforce  key  teaching  points. 

This  process  was  repeated  two  more  times.  The  major  difference  between  each  of  the 
three  iterations  was  that  the  profiles  presented  for  each  practical  exercise  were  each  designed  to 
highlight  different  instructional  objectives,  e.g.,  how  to  write  negative  bullets.  In  summary,  the 
instructional  cycle  captured,  conceptually,  the  essence  of  contrasting  cases/invention,  i.e., 
comparison  of  multiple  examples  (differing  profiles),  invention  (writing  bullets),  followed  by 
small  group  discussion,  feedback,  and  follow-on  lecture  highlighting  key  points  in  each  practical 
exercise.  Following  the  last  exercise  and  lecture,  the  PCI  students  completed  the  posttest,  and 
the  Post  Training  Questionnaire. 

Experimental  condition:  DI.  Students  arrived  at  one  of  the  ALC  small  group 
classrooms  and  were  briefed  on  the  general  research  objectives.  They  then  completed  the  pretest 
and  Demographic  Questionnaire. 

The  class  was  then  divided  into  two-man  teams.  The  instructor  first  addressed  the 
importance  of  having  an  evaluation  system  and  the  role  of  the  rater.  He  then  provided  a  45 
minute  lecture  in  which  he  reviewed  the  different  parts  of  the  NCOER  and  gave  guidelines  for 
Parts  I-III  (administrative  data,  authentication,  duty  description,  respectively),  Part  IV  -  Anny 
values  and  emphasized  the  critical  values  for  the  subordinates  (honor  and  integrity).  He  then 
provided  rules  for  constructing  bullet  comments.  Finally,  he  discussed  the  specific  sections 
listed  under  Part  IV  (i.e.,  competence,  leadership,  training,  responsibility  and  accountability). 

Following  the  lecture,  the  instructor  distributed  NCOER  materials  used  in  the  PCI 
module  with  the  major  difference  being  that  student  teams  received  information  on  only  one 
fictitious  Soldier  at  a  time.  Student  teams  read  through  the  materials  and  then  wrote  bullets  for 
this  fictitious  Soldier  on  specific  Anny  values,  attributes,  skills,  and  actions  on  acetate  sheets. 

As  with  the  PCI  module,  the  instructor  collected  the  sheets  and  displayed  each  sheet  on  an 
overhead  projector  for  the  whole  class  to  view.  The  class  provided  feedback  on  how  the  bullets 
could  be  improved.  The  instructor  provided  additional  feedback,  led  the  small  group  discussion, 
and  then  re-emphasized  key  teaching  points.  This  process  was  repeated  two  more  times.  Each 
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practical  exercise  was  different  and  designed  to  provide  examples  supporting  specific 
instructional  objectives.  In  summary,  the  design  of  the  DI  module  differed  from  the  PCI  module 
by  starting  with  an  extensive  lecture  followed  by  three  practical  exercises.  These  exercises  were 
less  complicated  than  the  PCI  exercises  since  each  involved  only  one  fictitious  Soldier  profile. 
Feedback  and  small  group  discussion  followed  each  exercise.  Following  the  last  exercise,  the  DI 
students  completed  the  posttest  and  the  Post  Training  Questionnaire. 

Control  condition.  Students  in  the  control  condition  had  previously  completed  the  then- 
current  1  IB  ALC  4-hour  module  covering  NCOER  preparation  and  Anny  writing.  They 
assembled  at  one  of  the  ALC  small  group  classrooms  and  were  briefed  on  the  general  research 
objectives.  They  then  completed  the  same  posttest  that  was  administered  to  the  DI  and  PCI 
students. 

Training  Efficiency 

For  purposes  of  this  project,  “efficiency”  refers  to  the  cost  of  a  training  alternative  that 
satisfies  the  tenninal  objectives  of  a  block  of  instruction.  Objective  measures  of  efficiency 
(MOEs)  were  developed  for  each  of  the  cost  factors  identified  through  the  PRM. 

Three  distinct  courses  of  action  (COA)  were  assessed  using  the  MOEs.  The  base 
situation  consisted  of  the  current  block  of  training  used  by  the  NCOA.  This  included  the 
materials  (produced  by  the  NCOA)  used  by  a  single  instructor,  mainly  lecturing  to  the  entire 
class  of  students.  The  second  situation  incorporated  instructional  materials  developed  for 
presentation  by  small  group  leaders  using  the  DI  approach.  The  third  situation  was  similar  to  the 
second,  but  was  designed  and  structured  for  presentation  using  the  invention/contrasting  cases 
instructional  (PCI)  approach. 

People  MOEs.  The  People  MOEs  refer  to  human  capital  requirements.  Broadly,  a  COA 
is  more  efficient  to  the  extent  it  requires  fewer  personnel  to  implement. 

Instructor-student  ratio.  In  general,  given  the  costs  associated  with  maintaining 
instructional  personnel,  a  lower  instructor-to-student  ratio  is  preferred. 

Recruiting  and  training  instructors.  In  general,  a  COA  is  more  efficient  to  the  extent  it 
minimizes  the  costs  of  identifying  and  preparing  instructors.  There  are  tradeoffs  to  be 
considered  within  this  MOE:  in  some  circumstances,  it  may  be  possible  to  identify  and  engage 
more  experienced  instructors  who  may  need  less  training  in  either  instructional  methods  or 
subject  domain  than  would  less  experienced  instructors. 

Technology  MOEs.  These  can  include  many  aspects  that  support  a  given  program  such 
as  applications,  infrastructure,  or  other  services.  This  analysis  highlighted  the  differences  among 
the  COA  based  on  two  factors:  training  site  and  training  aids. 

Number  and  size  of  classrooms.  To  the  extent  that  they  involve  dividing  a  course  into  a 
few  large  classes  or  into  several  smaller  class  sizes,  COA  may  differ  in  resourcing  requirements. 
In  general,  as  number  of  classrooms  increases,  the  classroom  common  infrastructure  costs 
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(display  capability,  instructor  support  items,  etc)  also  increase.  If  the  COA  also  involves 
construction  of  new  training  facilities,  the  life  cycle  cost  differential  between  large  and  small 
classrooms  must  also  be  considered. 

Training  aids.  In  the  context  of  this  work,  training  aids  include  training  material  and 
technology  above  and  beyond  the  classroom  common  infrastructure.  Examples  are  typically 
media  related,  such  as  audiovisual  equipment. 

Other  Fixed  Assets  MOEs.  These  include  any  other  physical  resource  used  in  a  COA. 

Paper  handouts  and  reference  materials.  COA  can  vary  in  physical  materials,  such  as 
handouts  and  reference  material,  required  to  conduct  training.  Some  of  these  materials  may  be 
re-used  from  class  to  class  and  replaced  only  periodically,  while  others  may  be  provided  to  the 
students  as  “take  away”  material  and  must  be  replaced  for  each  class. 

Development  of  appropriate  instructional  materials.  There  is  a  one-time  expenditure  of 
time  and  material  to  develop  training  matter  for  a  module,  and  there  is  a  smaller,  but  important, 
continuing  expenditure  to  maintain  that  training  matter.  There  is  also  the  possibility  that 
developing  training  matter  for  a  COA  may  require  specialized  (and  more  expensive)  expertise 
for  the  training  developer. 

Time  to  conduct  training  to  standard.  In  general,  the  more  rapidly  a  COA  will  result  in 
Soldiers  trained  to  standard  for  a  task,  the  more  efficient  that  COA  is.  In  the  case  of  the  NCOER 
preparation  training  module,  4  hours  were  already  allocated  for  training.  If  a  COA  were  to 
require  significantly  more  or  significantly  less  time  for  Soldiers  to  train  to  standard,  then  the 
relative  efficiencies  would  become  a  greater  consideration. 

Transportation  costs.  If  a  COA  involves  Soldiers’  using  a  different,  second  training 
venue,  then  transportation  resources  may  become  a  factor.  These  transportation  costs  include 
both  the  monetary  cost  of  movement  and  the  training  “dead”  time  involved  while  Soldiers  are  in 
transit. 


Results 

Overall  Findings 

For  this  research,  we  measured  participants’  performance  on  well-defined  (WD)  tasks  in 
terms  of  their  score  on  the  pretest  and  posttest  sections  relating  to  the  Rules  and  Principles  for 
Writing  NCOER  Bullets  and  the  Rules  and  Principles  of  the  NCOER  Evaluation  Reporting 
System.  Items  from  these  sections  addressed  the  participants’  understanding  of  well-defined 
rules  for  the  completion  of  NCOERs  and  have  objectively  correct  answers.  The  well-defined 
task  score  was  based  on  the  percentage  of  the  total  points  accrued  by  the  participant  on  these  two 
sections.  The  maximum  possible  WD  score  was  49%  on  the  pretest  and  52%  on  the  posttest. 

Similarly,  we  measured  participants’  perfonnance  on  ill-defined  (ID)  tasks  in  terms  of 
their  score  on  the  pretest  and  posttest  section  relating  to  NCOER  knowledge  and  bullet  writing. 
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These  items  required  the  participant  to  review  a  fictitious  Soldier’s  performance  record  and 
identify  a  bullet  that  accurately  and  effectively  reflected  that  performance.  The  rest  of  the  ill- 
defined  task  items  were  in  a  short  answer  fonnat  in  which  the  participant  is  asked  to  produce  a 
novel  bullet  based  on  the  fictitious  Soldier’s  perfonnance.  These  items  were  subjectively  rated 
on  three  attributes:  Beginning,  Concise,  and  Substantive  (see  Table  5).  The  ill-defined  task 
score  was  based  on  the  percentage  of  the  total  points  accrued  by  the  participant  on  this  section. 
The  maximum  possible  ID  score  was  5 1%  on  the  pretest  and  48%  on  the  posttest. 


Table  5.  Scoring  Rubric  for  Ill-defined  Task  Items  Using  the  Short  Answer  Format 


Attribute 

Max.  Points 

Description 

Beginning 

2 

Bullet  starts  with  or  contains  a  verb  in  past  tense  to  address 
perfonnance  or  contributions,  or  begins  with  a  personal 
pronoun.  The  bullet  is  in  active  voice 

Concise 

2 

Bullet  is  concise,  short  and  to  the  point;  not  longer  than  two 
lines  (approximately  185  characters  and  spaces  in  Part  I.V.a. 
and  approximately  165  characters/spaces  for  Part  IV.  b.-f. 

Substantive 

3 

Bullet  provides  a  quantitative  or  comparative,  substantiated 
comment  on  the  rated  NCO’s  performance,  results, 
achievements,  deficiencies,  or  shortcomings  supporting  the 
ARMY  VALUES  or  VALUES/NCO  RESPONSIBILITIES 
rating. 

Figure  3  illustrates  the  overall  effectiveness  measurements  for  the  DI  and  PCI  students. 

To  evaluate  participant  learning  as  affected  by  learning  condition,  a  three-way  (2x2x2)  mixed 
factor  analysis  of  variance  (ANOVA)  was  conducted  on  task  score.  The  independent  variables 
included  one  between  groups  variable,  experimental  condition,  with  two  levels  (DI,  PCI)  and  two 
within  subjects  variables,  time  and  task  definition,  with  two  levels  each  (pretest/posttest  and 
well-defined/ill-defined,  respectively). 

The  Box’s  M  test  indicated  no  evidence  to  suggest  that  the  homogeneity  of  variance 
assumption  was  violated  (Box’s  M=  12.484,  F(10,15,  489)=1 . 1 53,  =  0.318).  There  was  a 

significant  main  effect  of  time  (77i>56=  134.7 16,  MSE= 0.008,  p  <  0.001).  Overall,  the  average 
score  on  the  posttest  (75%)  was  greater  than  on  the  pretest  (58%)  regardless  of  condition  or  task 
definition.  Simple  main  effects  analysis  demonstrated  that  the  pretest  and  posttest  scores  were 
significantly  different  for  both  the  well-defined  (Fi!56=79.695,  MSTsMhOOF/^O.OOl)  and  ill- 
defined  tasks  (Fi,56=1  10.8  8  8,  MSE= 0.012,  p<0.001).  There  were  no  significant  interactions,  and, 
in  particular,  no  interaction  between  experimental  condition  and  type  of  task.  (Figure  3). 
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Figure  3.  DI  and  PCI  students  showing  overall  difference  between  pretest/posttest  and 
between  ID/WD 


Considering  previous  experience  as  a  rater  in  preparing  NCOERs  (cf.  Table  4)  to  be  an 
indicant  of  expertise,  correlations  between  posttest  scores  and  number  of  NCOERs  rated  for 
well-defined  and  for  ill-defined  items  for  DI  students  and  PCI  students  were  all  calculated.  None 
of  the  four  correlations  were  significant. 

To  determine  how  well  the  two  experimental  training  modules  faired  relative  to  the 
control  group,  a  two-way  (2x3)  mixed  factor  ANOVA  was  conducted  on  posttest  task  score.  The 
independent  variables  included  one  between  groups  variable,  condition,  with  three  levels  (DI, 
PCI,  and  Control)  and  one  within  subjects  variable,  task  definition,  with  two  levels  (well-defined 
and  ill-defined).  There  was  a  significant  main  effect  of  condition  (62,84=5.558,  MSE=0.020, 
p<0.005).  Additionally,  there  was  a  significant  interaction  between  task  definition  and  condition 
(E3  84=13. 75 1,  MSE=0.0  1 2,  p<0.001).  Simple  main  effects  analysis  demonstrated  that  the 
posttest  scores  for  the  well-defined  tasks  did  not  differ  significantly  (F2, 84=0. 169,  MSE= 0.020, 
/?=().  845),  but  the  posttest  scores  for  ill-defined  tasks  were  significantly  different  (62, 86=23. 43  8, 
MSE=0.0 1 2,  /;<(). 00 1 ).  Post  hoc  Tukey  tests  (at  p<0.05)  were  conducted  to  explore  this  effect 
and  found  that  on  average  the  posttest  score  for  ill-defined  tasks  of  the  control  group  (62%)  was 
significantly  lower  than  that  of  participants  in  either  the  DI  (78%)  or  PCI  (79%)  conditions  (See 
Figure  4). 
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Figure  4.  Comparison  of  the  posttest  scores  for  well-defined  (WD)  and  ill-defined  (ID) 
tasks  by  experimental  condition. 


Pre/Post  Differences  in  Level  of  Understanding 

The  participants  were  asked  to  rate  on  a  five  point  Likert-type  scale,  ranging  from  zero 
(“Do  Not  Understand”)  to  5  (“Could  Teach  This  to  Others”),  their  level  of  understanding  before 
and  after  instruction  on  several  activities  related  to  completing  NCOER  forms:  (a)  Conducting 
meaningful  counseling  and  documenting  results,  (b)  using  the  NCOER  support  form  to  fully 
document  performance,  (c)  including  meaningful  bullets  that  accurately  document  performance, 
and  (d)  using  supporting  documentation  when  developing  ratings  and  meaningful  bullets.  A 
two-way  (2x2)  mixed  multivariate  analysis  of  variance  was  conducted  on  the  ratings  for  the 
activities.  The  independent  variables  included  one  between  groups  variable,  experimental 
condition,  with  two  levels  (DI  and  PCI)  and  one  within  subjects  variable,  time,  with  two  levels 
(pretest  and  posttest).  There  was  not  a  significant  main  effect  for  condition  for  any  of  the 
activities  (^4,52=0. 322,  /;=().  862),  but  there  was  a  significant  and  interpretable  main  effect  of  time 
(F4, 52=51.100, /K0.001).  Participants  in  both  experimental  conditions  felt  that  their 
understanding  increased  significantly  after  instruction  for  all  activities.  The  results  revealed  a 
significant  interaction  between  time  and  experimental  condition  (F4  52=3.0 1 5,  /;<(). 026).  Post  hoc 
univariate  ANOVAs  for  each  activity  revealed  that  the  interaction  effect  between  time  and 
experimental  condition  was  only  significant  for  “Conducting  Meaningful  Counseling  and 
Documenting  Results”  (Ej^M  1.526,  MSfi=0.363,/K0.001).  Participants  in  the  PCI  condition 
reported  a  greater  increase  in  their  understanding  of  this  activity  as  a  result  of  the  instruction  than 
did  the  DI  condition.  This  finding  is  due  to  the  two  experimental  conditions  reporting  a 
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significantly  different  understanding  of  this  activity  prior  to  instruction  {t^=2. 1 04,  /;<(). 04; 
Figure  5). 


Could  Teach 
This  to  Others 


Understand 
Very  Well 


Got  It 


✓ 

✓ 


Understand 

Somewhat 


Pre 

Post 

Pre 

Post 

DI 

PCI 

----Conducting  Meaningful  Counseling  &  Documenting  Results 
—  —  Using  NCOER  Support  Form  to  Fully  Document  Performance 

Including  Meaningful  Bullets  that  Accurately  Document  Performance 
. UsingSupporting  Documentation  in  Developing  Ratings  &  Meaningful  Bullets 


Figure  5.  Mean  ratings  for  level  of  understanding  of  activities  before  and  after  instruction 
by  experimental  condition. 


Student  Perceptions  of  Training  Value 

To  evaluate  the  effect  that  the  two  instructional  methodologies  had  on  the  participants’ 
perception  of  the  value  of  the  NCOER  preparation  instruction,  independent  groups  T-tests  were 
conducted  on  the  eleven  posttest  questionnaire  items  addressing  students’  valuation  of  the 
training.  The  independent  variable  was  the  between  groups  variable,  experimental  condition, 
with  two  levels  (DI  and  PCI).  Ratings  for  all  categories  were  higher  for  PCI  than  DI  with 
significant  differences  (all  favoring  PCI)  found  for  Item  5,  /(54)=2. 1 74,  p  <.05;  Item  7, 
t(55)=2.256,/K.05;  and  Item  10,  t(55)=2.491,/?<.05.  The  items  appear  in  Table  6  and  their 
mean  ratings  appear  in  Figure  6. 
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Table  6.  Items  Relating  to  Perceptions  of  Value  of  Training 


Category  -  “The  instruction...” 

1  Gave  me  a  much  better  understanding  of  the  NCOER  process. 

2  Prepared  me  to  perform  my  duties  as  a  rater. 

3  Prepared  me  to  develop  more  meaningful  and  accurate  NCOERs. 

4  The  time  devoted  to  explaining  concepts  was  adequate. 

5*  Covered  issues  and  nuances  in  the  NCOER  process  that  were  very  helpful. 

,  Provided  valuable  insights  on  how  to  effectively  develop  appropriate  bullets  for  an 
b  NCOER. 

Improved  my  ability  to  write  an  NCOER  that  accurately  represents  the  NCO’s 
performance. 

8  Provided  me  a  better  appreciation  for  accessing  and  using  Army  publications. 

9  Assisted  me  in  knowing  what  1  need  to  do  to  receive  an  “Excellent”  rating. 

10*  Idas  motivated  me  to  ensure  my  subordinates  receive  accurate  NCOERs. 

1 1  Has  motivated  me  to  ensure  that  my  NCOER  accurately  reflects  my  performance. 
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Figure  6.  Mean  ratings  for  perception  of  instructional  value  by  experimental  condition  and 
category. 


Institutional  Resources 

For  the  measures  of  efficiency  listed  in  Table  7,  approximate  resources  required  or 
expended  were  recorded.  It  should  be  noted  that,  because  the  resource  requirements  were  only  to 
be  characterized  instead  of  rigorously  quantified,  the  quantities  in  some  cases  may  range  from 
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approximate  to  anecdotal.  Three  distinct  courses  of  action  were  assessed  using  the  MOEs:  the 
current  situation,  the  DI  approach,  and  the  PCI  approach.  Some  historical  resource  requirements 
for  the  current  situation,  e.g.  ratio  of  preparation  hours  to  instructional  hours,  were  not  directly 
available  but  were  estimated  by  consensus. 

People  MOEs.  Broadly,  the  current  block  of  training  is  relatively  more  efficient  than 
both  the  DI  and  PCI  methodologies  across  the  dimensions  of  People  MOEs. 

Instructor-student  ratio.  Currently,  one  instructor  can  train  up  to  160  students 
simultaneously  in  a  mass  lecture  fonnat.  Both  the  DI  and  PCI  methodologies  require  a  small- 
group  (i.e.,  approximately  16  students)  approach. 

Recruiting  and  training  instructors.  The  current  training  can  be  completed  by  one 
person  with  limited  knowledge  of  the  content  using  Microsoft  PowerPoint  (PPT)  slides  to 
present  material.  On  the  other  hand,  the  instructors  for  DI  and  PCI  require  a  greater  level  of 
class  content  knowledge.  The  practical  exercises  (PEs)  require  close  interaction  with  students.  It 
is  estimated  that  the  instructors  will  generally  take  at  least  two  (for  DI)  to  four  (for  PCI)  times  as 
much  training  and  preparation  time  to  administer  the  class.  The  PCI  methodology,  in  particular, 
requires  a  high  level  of  expertise  in  the  content  area  and  the  ability  for  the  instructor  to  go  “off- 
script”  in  response  to  the  needs  of  the  class.  Replacing  and  preparing  small  group  instructors 
(SGIs)  with  sufficient  depth  of  knowledge  to  present  material  and  to  facilitate  the  PEs  using  the 
PCI  approach  is  much  more  complicated  and  demanding  than  the  two  other  approaches. 

Technology  MOEs.  The  Technology  MOEs  can  include  many  aspects  that  support  a 
given  program  such  as  applications,  infrastructure,  or  other  services.  This  analysis  highlighted 
the  differences  among  the  instructional  methodologies  based  on  two  factors,  classrooms  and 
training  aids. 

Number  and  size  of  classrooms.  The  current  training  is  conducted  in  a  single  room, 
large  enough  to  seat  up  to  160  students.  The  DI  and  PCI  training  require  10  sixteen-person 
classrooms. 

Training  aids.  In  the  current  training,  a  single  projection  system  is  required,  with 
sufficient  viewing  screens  properly  positioned  for  all  students  to  see  the  screens.  Presentation  is 
in  PPT  so  a  computer  device  necessary  to  show  the  slides  is  required.  Handouts  and  exercises 
are  extremely  limited  so  desk  space  for  each  student  is  not  needed. 

In  both  the  DI  and  PCI  training  each  classroom  needs  to  have  a  projection  system  and 
screens  that  can  be  viewed  by  all  students.  Some  presentation  is  in  Microsoft  Powerpoint,  so  a 
computer  device  is  required  to  show  the  slides.  Handouts  and  exercises  are  extensive  so  desk 
space  for  each  student  is  required.  (Note:  These  capabilities  already  exist  and  are  used  for  other 
ALC  classes.) 

To  properly  conduct  the  DI  and  PCI  training,  a  method  must  be  available  for  students  to 
quickly  and  easily  compare  their  inputs  with  other  students  and  to  facilitate  discussion.  During 
pilot  sessions  this  was  accomplished  using  an  overhead  projector  with  prefonnatted  acetate 
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slides.  Other  alternatives  are  possible.  For  example,  students  could  each  move  to  a  central 
computer  tenninal  and  enter  their  input  which  could  then  be  projected  in  the  classroom.  This 
technique  would  require  additional  time  while  each  student  enters  input  before  progressing  into 
discussions.  Another  possibility  would  require  students  to  have  a  computer  device  that  could 
transmit/share  input  to  the  central  computer  tenninal  (e.g.,  e-mail,  shared-drive).  In  this 
approach,  each  student  would  prepare  input  and  send  it  to  the  central  computer  terminal  for 
projection.  Additionally,  the  DI  and  PCI  training  methods  require  approximately  50  pages  of 
reference  material  for  each  student.  These  items  can  be  recovered  at  the  end  of  the  class,  but 
would  need  to  be  stored  for  reuse  with  subsequent  classes.  Most  students  wanted  to  take 
reference  materials  with  them  for  future  use.  If  this  were  allowed,  reproduction  of  references 
would  become  a  recurring  cost  and  storage  space  would  not  be  required.  If  an  overhead 
projector  is  used  to  allow  student  sharing/comparing  of  input,  approximately  24  acetate  slides  are 
needed  for  each  SGI.  These  items  are  reusable,  but  would  need  to  be  replaced  periodically  due 
to  wear  and  tear. 

Other  Fixed  Assets  MOEs 

Paper  handouts  and  reference  materials.  The  physical  materials  such  as  handouts  and 
reference  material  required  to  conduct  the  current  training  amount  to  less  than  5  pages.  The  DI 
training  uses  approximately  125  pages  and  5  manila  folders,  and  the  PCI  training  uses  roughly 
170  pages  and  8  manila  folders.  For  both  DI  and  PCI,  about  80%  of  the  materials  could  be  saved 
and  reused  for  subsequent  classes.  This  would  require  instructors  to  ensure  students  do  not  mark 
on  papers.  Also,  instructors  would  need  to  recover  all  materials  at  the  end  of  class,  then  sort  and 
check  papers/folders  to  ensure  all  necessary  papers  are  retained  in  the  correct  packets. 

Development  of  appropriate  instructional  materials.  As  a  general  “rule-of-thumb”,  on 
average,  it  takes  about  10  hours  to  develop  one  hour  of  training.  This  is  not  documented,  but  is 
used  by  some  Anny  training  developers.  The  materials  for  the  current  training  are  already  well 
established  and  basically  require  no  preparation  prior  to  the  instruction  period.  For  this  effort, 
the  DI  training  required  approximately  25  hours  to  develop  each  hour  of  training  (a  total  of  100 
hours  for  the  4-hour  NCOER  class),  in  order  to  cover  the  material  to  the  desired  level  of 
proficiency  and  provide  sufficient  challenging  PEs.  The  PCI  training  required  approximately  40 
hours  to  develop  each  hour  of  training  (a  total  of  160  hours  for  the  4-hour  NCOER  class),  in 
order  to  cover  the  material  to  the  desired  level  of  proficiency  and  provide  sufficient  contrasting 
and  challenging  PEs. 

Time  to  conduct  training  to  standard.  The  current  POI  allots  4  hours  for  the  NCOER 
class.  Therefore,  the  DI  and  PCI  variants  were  constructed  to  generally  fit  within  this  available 
time.  According  to  ALC  cadre,  the  current  class  does  not  adequately  train  students  to  the  desired 
standard.  Training  to  standard  could  be  accomplished  within  the  allotted  4-hour  time  frame. 
However,  our  assessment  is  that  the  current  training  content  and  presentation  style  would  need  to 
be  altered  in  order  to  conduct  training  to  standard.  Based  on  pilot  training  sessions,  material  for 
the  DI  condition  was  presented  within  the  allotted  4-hours,  including  completion  of  the  post-test. 
The  proper  and  adequate  presentation  of  the  PCI  material,  which  includes  sufficient  time  to 
appropriately  conduct  the  variety  of  contrasting  cases,  would  take  about  5-hours,  including 
completion  of  the  post-test. 
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Transportation  costs.  When  the  research  was  perfonned,  all  students  had  to  be 
transported  to  or  provide  their  own  transportation  to  move  from  the  SGI  classrooms  to  a  separate 
facility  that  seated  the  entire  class  (up  to  160  students).  The  movement  time  also  added 
approximately  one  hour  to  the  training  time,  including  student  movement  from  SGI  rooms  to 
transport  location,  transport  to  alternate  site,  movement  into  large  facility,  and  the  reverse 
process  for  returning  to  the  SGI  classrooms.  Currently,  the  SGI  classrooms  and  the  lecture 
facility  are  located  within  the  same  building  complex.  With  the  current  building  complex,  there 
are  no  significant  transportation  differences  across  the  three  courses  of  action. 

Institutional  resources  required  for  this  specific  block  of  instruction  are  summarized  in 
Table  7. 


Table  7.  Assessment  of  Measures  of  Efficiency  for  Each  Instructional  Methodology 


Measure 

Current 

DI 

PCI 

People  MOEs 

Instructor  Student  Ratio 

1:160 

1:16 

1:16 

Recruiting  &  Training  Instructors 

Easy 

Difficult 

More  difficult 

Technology  M< 

OEs 

No.  of  classrooms 

1  large 

10  small 

10  small 

Training  Aids 

Projection 

capability 

About  50  pages  of 
material,  projection 
capability 

About  50  pages  of 
material,  projection 
capability 

( 

Dther  Fixed  Asset 

MOEs 

Handouts  and  Reference 

Materials  per  Student 

About  5  pages 

About  125  pages 

About  170  pages 

Instructional  Materials  Develop¬ 
ment  (prep  hrs:instructional  hrs) 

10:1 

25:1 

40:1 

Time  to  Conduct  Training  to 
Standard 

4  hr 

4  hr 

5  hr 

Transportation  Costs 

Negligible 

Negligible 

Negligible 

Discussion 

Overview 

ALC  2015  recommends  a  shift  towards  more  learner-centered,  experiential  instructional 
methodologies  under  the  overarching  assumption  that  in  many  cases  this  shift  will  result  in  more 
effective  training.  This  research  sought  to  examine  the  effects  of  such  a  shift  for  a  typical  Army 
training  module.  The  research  was  situated  in  a  typical  on-going  Army  institutional  training 
environment,  utilized  typical  Army  students,  and  employed  typical  Army  assessments  of 
learning.  Thus  it  was  expected  that  any  statistically  significant  results  would  likely  also  be 
operationally  significant,  and,  more  importantly,  that  statistically  non-significant  results  would 
have  no  operational  significance  to  the  Army.  Also,  given  that  the  research  was  situated  in  a 
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typical  Army  training  environment,  categories  of  training  resource  elements  would  likely  be 
similar  to  those  to  be  found  in  typical  Army  training. 

The  research  evaluated  the  relative  effectiveness  of  a  PCI  module  to  that  of  a  DI  module 
and  the  training  the  students  currently  receive  in  ALC.  Previous  studies  suggest  that  PCI  should 
be  better  suited  to  training  abstract  and  ill-defined  tasks  and  DI  should  be  a  more  effective 
methodology  for  well-defined,  procedural  tasks.  Both  the  PCI  and  DI  modules  were  found  to 
significantly  improve  student  scores  on  tasks  related  to  NCOER  completion  from  pretest  to 
posttest.  The  results  did  not  support  our  hypotheses  of  differential  effectiveness  of  the  PCI 
instructional  methodology  compared  to  DI  for  the  more  ill-defined  NCOER  completion  tasks. 
Despite  similar  performance  on  the  measures,  there  were  significant  differences  in  the  students’ 
perceptions  of  their  training. 

The  absence  of  the  anticipated  significant  interaction  between  instructional  approach  and 
instructional  domain  has  two  alternative  implications:  (1)  there  is  no  significant  differential 
effect  of  instructional  approach  or  (2)  there  actually  is  a  significant  effect,  but  the  present 
research  failed  to  detect  it.  These  two  alternatives,  along  with  considerations  of  required  training 
resources,  are  discussed  in  some  detail  below. 

No  Difference  between  DI  and  PCI  Approaches? 

There  has  been  a  widespread  caution  (cf.  Kirschner,  Sweller  &  Clark,  2006;  Mayer, 
2004)  that  PCI  approaches  are  at  best  no  more  effective  than  are  DI  approaches.  More  recently, 
Benson  (2012)  concludes  that  there  is  no  compelling  empirical  evidence  to  support  mandating 
PCI  as  a  superior  pedagogy,  but  on  a  positive  note  concludes  that,  since  it  appears  to  be 
equivalent  to  DI,  PCI  might  be  adopted  for  reasons  other  than  training  effectiveness.  Onyon 
(2012)  also  concludes  that  compelling  evidence  for  adoption  of  PCI  remains  undetermined  and 
further  suggests  that  more  effort  be  made  to  explore  implications  of  some  of  the  theoretical 
underpinnings  of  PCI. 

The  result  of  this  particular  research  is  certainly  congruent  with  previous  findings  of 
indetenninate  differences.  The  equal  end-of-course  perfonnances  of  the  DI  and  the  PCI  classes 
would  indicate  no  operationally  significant  difference  between  these  two  instantiations  of  the 
approaches.  The  inference  could  be  made  that,  if  instruction  has  been  well  crafted,  either  as  per 
DI  principles  or  as  per  PCI  principles,  the  result  will  be  effective,  at  least  as  typically  assessed  by 
the  Army. 

A  Real,  but  Undetected  Difference? 

On  the  other  hand,  there  remains  the  possibility  that  there  was  some  difference  in 
learning,  but  that  the  difference  was  not  captured  by  this  research.  Indications  that  the  two 
instructional  approaches  were  at  some  level  not  completely  equivalent  come  from  differences  in 
the  DI  and  PCI  students’  perceived  understanding  of  the  subject  domain  and  their  perceived 
value  of  the  training. 

Students  in  the  PCI  condition  rated  their  prior  understanding  of  the  importance  of 
conducting  counseling  and  documenting  the  results  significantly  lower  than  did  the  DI  students. 
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Because  their  ratings  for  prior  and  current  understanding  were  both  evaluated  after  the  training 
module,  it  is  possible  that  the  PCI  students  gained  a  different  appreciation  of  their  initial  lack  of 
knowledge. 

Student  perceptions  of  the  training  module’s  value  were  more  positive  in  the  PCI 
condition.  It  is  possible  that  the  students  in  the  PCI  condition  viewed  the  instruction  more 
favorably  for  several  reasons.  Increased  PCI  workload  on  exercises  relative  to  DI  may  have 
influenced  PCI  students’  perceptions.  The  greater  cognitive  demands  placed  on  PCI  students 
may  have  led  students  to  believe  that  more  work/effort  expended  in  training  produced  better 
understanding  of  material  and  higher  levels  of  perceived  perfonnance  or  competency. 

There  is  the  possibility  that  the  primary  dependent  measures  themselves  used  in  this 
research  somehow  are  insensitive  to  real  differences  between  the  two  instructional  models.  For 
example,  additional  alternative  or  complementary  measures,  such  as  response  latency,  response 
confidence,  etc.  might  be  sensitive  enough  to  detect  significant  differences  between  DI  and  PCI 
approaches. 

Additionally,  there  is  the  possibility  that  PCI-instructed  students  might  out-perfonn  DI- 
instructed  students  on  a  subsequent  far  transfer  task,  such  as  the  fairly  common  Anny 
administrative  task  of  drafting  citations  for  personal  awards.  Schwartz  and  Martin  (2004)  found 
such  an  advantage  for  contrasting  cases/invention.  However,  in  later  work  Schwartz,  Chase, 
Oppezzo,  and  Chin  (2011)  examined  the  effectiveness  of  contrasting  cases  when  some  students 
are  presented  the  to-be-learned  concepts  prior  to  being  presented  the  contrasting  cases  and  others 
presented  the  concepts  after  being  presented  the  contrasting  cases.  Schwartz  et  al.  found  that, 
although  both  groups  of  students  perfonned  approximately  equivalently  on  the  to-be-leamed 
task,  students  who  were  presented  the  to-be-leamed  concepts  after  the  contrasting  cases 
performed  better  on  a  related  far  transfer  task.  Schwartz  et  al.  conclude  that,  when  students 
experience  the  initial  “inventing”  of  the  to-be-learned  concepts  before  being  formally  presented 
the  concepts,  the  underlying  deep  structures  are  better  learned  and,  consequently,  far  transfer  is 
made  easier.  In  the  present  research,  given  that  70%  of  the  students  had  already  participated  in 
preparing  NCOER  bullets  (cf.  Table  3)  and,  thus,  presumably  had  some  concept  of  the  NCOER 
preparation  task,  and  given  that  they  had  completed  the  pretest,  it  is  unlikely  that  the  PCI  group 
experienced  any  appreciable  “inventing”  during  instruction. 

The  likelihood  that  students’  prior  experience  or  expertise  with  preparing  NCOERs 
significantly  affected  their  learning  was  not  supported.  The  lack  of  significant  correlation 
between  the  number  of  NCOERs  students  had  previously  prepared  and  their  performance  on  the 
posttest  would  indicate  that  previous  experience  had  no  appreciable  effect  on  terminal  learning 
level. 


The  foregoing  considerations  suggest  areas  for  further  research  into  the  effectiveness  of 
PCI  methods.  However,  for  purposes  of  the  present  research,  the  fact  remains:  for  typical  Anny 
students  in  a  typical  Anny  environment  using  typical  Army  training  assessment  measures,  there 
was  no  operationally  significant  difference  found  between  the  two  approaches. 
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Can  Required  Resources  Make  a  Difference? 


With  regards  to  training  effectiveness,  the  then  current  NCOER  preparation  training  was 
less  effective  than  either  the  DI  or  the  PCI  training,  and  DI  and  PCI  were  equivalently  effective. 
With  regards  to  training  resources,  this  finding  sets  up  two  comparisons  among  the  training 
methods:  (a)  current  method  vs.  DI/PCI  and  (b)  DI  vs.  PCI. 

Current  method  vs.  DI/PCI.  Although  the  then  current  method  was  found  less  effective 
than  DI/PCI,  there  is  the  possibility  that  DI/PCI  would  be,  relative  to  the  current  method,  much 
too  resource  intensive  to  warrant  developing  and  sustaining  a  new  method  of  instruction.  A 
cursory  inspection  of  Table  7  reveals  that,  of  the  three,  the  then  current  method  is  least  resource 
intensive.  This  then  raises  the  practical  question  for  training  developers:  is  the  gain  in 
effectiveness  attributable  to  the  DI/PCI  methods  (cf.  Figure  2)  worth  the  additional  resources 
required  to  implement  the  DI/PCI  methods?  In  particular,  is  the  additional  skill  at  ill-defined 
aspects  of  preparing  NCOERs  worth  the  additional  resources?  A  quantitatively  supported 
answer  to  these  questions  is  outside  the  scope  of  the  present  effort,  but  these  are  examples  of 
issues  that  may  arise  and  the  data  that  may  be  required  to  address  them  when  training 
effectiveness  is  traded  off  against  training  efficiency. 

DI  vs.  PCI.  If  the  gain  in  effectiveness  due  to  the  DEPCI  methods  outweighs  the 
additional  resources  required  to  implement  them,  then  the  question  arises  as  to  whether  there  are 
requirements  differences  of  a  magnitude  to  recommend  one  method  over  the  other.  As  was 
anticipated,  the  PCI  method  required  more  resources  in  instructor  training  and  materials 
development  than  did  the  DI  method  (cf.  Table  7).  However,  there  was  no  off-setting  increase  in 
effectiveness  associated  with  the  PCI  method.  Thus,  unless  the  students’  increase  in  perceived 
understanding  and  increase  in  perceived  value  more  than  offsets  the  resource  differential,  in  a 
decision  to  adopt  either  the  DI  or  the  PCI  method,  the  training  developer  would  choose  the  DI 
method. 


Conclusions 

The  results  of  this  research  support  the  cautions  (e.g.,  Kirschner  et  ah,  2006;  Mayer, 

2004)  that  have  been  raised  concerning  indiscriminant  application  of  PCI  methods.  Although  the 
ALM  prescribes  that  the  Anny  embrace  “context-based,  collaborative,  problem-centered 
instruction,”  these  results  indicate  that,  in  at  least  some  cases,  there  may  be  no  benefit  of 
problem-centered  instruction  over  direct  instruction.  This  research  indicates  that  it  is  possible 
using  best  practices  within  a  typical  Anny  institutional  training  environment  to  design  both  PCI 
and  DI  that  are  equivalent  in  their  training  effectiveness,  as  typically  measured  in  that 
institutional  environment.  Additionally,  the  research  shows  the  equivalency  can  hold  regardless 
of  whether  the  subject  domain  is  well-defined  or  ill-defined. 

However,  just  as  PCI  should  not  be  indiscriminately  applied,  so  should  the  results  of  this 
research  not  be  indiscriminately  applied  to  all  training.  The  results  by  no  means  indicate  that 
well  executed  DI  and  PCI  are  equivalent  -  the  results  should  be  interpreted  as  a  caution  for 
training  developers  that  in  many  cases  any  relative  effectiveness  advantage  to  either  method  may 
not  be  operationally  relevant. 
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This  research  also  involved  a  high-level  examination  of  the  relative  costs  of  DI  and  PCI 
in  terms  of  required  resources.  The  results  indicated,  as  anticipated,  PCI  to  require  more 
resources  for  development  and  for  sustainment.  This  resource  differential  will  likely  hold  across 
any  Army  comparison  of  DI  and  PCI,  and  training  developers  should  take  into  consideration  the 
difference  in  resource  requirements  when  making  any  decision  between  implementing  DI  or  PCI. 


27 


28 


References 


Bell,  B.  S.,  &  Kozlowski,  S.  W.  J.  (2008).  Active  learning:  Effects  of  core  training  design 

elements  on  self-regulatory  processes,  learning,  and  adaptability.  Journal  of  Applied 
Psychology,  93,  296-316. 

Blume,  B.  D.,  Ford,  J.  K.,  Baldwin,  T.  T.,  &  Huang,  J.  L.  (2010).  Transfer  of  training:  A  meta- 
analytic  review.  Journal  of  Management,  36,  1065-1105.. 

Bransford,  J.  D.,  Franks,  J.  J.,  Vye,  N.  J.,  &  Sherwood,  R.  D.  (1989).  New  approaches  to 
instruction:  Because  wisdom  can’t  be  told.  In  S.  Vosniadou  &  A.  Ortony  (Eds.), 
Similarity  and  Analogical  Reasoning  (pip.  470-497).  New  York:  Cambridge  University 
Press. 

Chen,  G.,  Gully,  S.  M.,  &  Eden,  D.  (2001).  Validation  of  a  new  general  self-efficacy  scale. 
Organizational  Research  Methods,  4,  62-83. 

Cianciolo,  A.  T.,  Grover,  J.,  Bickley,  W.  R.,  &  Manning,  D.  (2011).  Problem-based  learning: 

Instructor  characteristics,  competencies,  and  professional  development  (Research  Report 
1936).  Arlington,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences.  (DTIC  No.  ADA535416) 

Clark,  R.  E.  (2004).  Design  document  for  a  guided  experiential  learning  course.  Document 
retrieved  from  http://www.cogtech.usc.edu/publications/clark  gel.pdf 

Dollinger,  S,  J.  &  DiLalla,  D.  L.  (1996).  Cleaning  up  data  and  running  preliminary  analyses.  In 
F.  Leong  &  J.  Austin  (Eds.),  The  Psychology  Research  Handbook  (pp.  167-176). 
Thousand  Oaks,  CA:  Sage  Publications. 

Gosling,  S.  D.,  Rentfrow,  P.  J.,  &  Swann,  W.B.  (2003).  A  very  brief  measure  of  the  Big-Five 
personality  domains.  Journal  of  Research  in  Personality,  37,  504-528. 

Hmelo-Silver,  C.  E.,  (2004).  Problem-based  learning:  What  and  how  do  students  learn? 
Educational  Psychology  Review,  16,  235-266. 

Hmelo-Silver,  C.  E.,  Duncan,  R.  G.,  &  Chinn,  C.  A.  (2007).  Scaffolding  and  achievement  in 

problem-based  and  inquiry  learning:  A  response  to  Kirschner,  Sweller,  and  Clark  (2006). 
Educational  Psychologist,  42,  99-107. 

Merrill,  M.  D.,  (2009).  First  principles  of  instruction.  In  C.  M.  Reigeluth  &  A.  Carr  (Eds.), 

Instructional  design  theories  and  models:  Building  a  common  knowledge  base  (Vol.  III). 
New  York:  Routledge  Publishers. 


29 


Orvis,  K.  A.,  Horn,  D.  B.,  &  Belanich,  J.  (2006).  Videogame-based  training  success:  The 

impact  of  student  characteristics  -  Year  2  (Technical  Report  1188).  Arlington,  VA:  U. 

S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences.  (DTIC  No.  ADA 
457396) 

Pleban,  R.  J.,  Graves,  T.  R.,  Miller,  M.  L.,  Donigian,  A.  M.,  Branciforte,  J.  V.,  &  Matthews,  M. 
D.  (2011).  Evaluating  a  problem-based  learning  strategy  for  enhancing  ethical 
awareness  in  negotiation  (Research  Report  1938).  Arlington,  VA:  U.  S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences.  (DTIC  No.  ADA535714) 

Pleban,  R.  J.,  Vaughn,  E.  D.,  Sidman,  J.,  Geyer,  A.,  &  Semmens,  R.  (2011).  Training  platoon 

leader  adaptive  thinking  skills  in  a  classroom  setting  (Research  Report  1948).  Arlington, 
VA:  U.S.  Anny  Research  Institute  for  the  Behavioral  and  Social  Sciences.  (DTIC  No. 
ADA544978) 

Schraw,  G.  &  Dennison,  R.S.  (1994).  Assessing  metacognitive  awareness.  Contemporary 
Educational  Psychology,  19,  460-475. 

Schwartz,  D.  L.,  &  Black,  J.  B.  (1996).  Shuttling  between  depictive  models  and  abstract  rules: 
Induction  and  fallback.  Cognitive  Science:  A  Multidisciplinary  Journal,  20,  457-497. 

Schwartz,  D.  L.,  &  Bransford,  J.  D.  (1998).  A  time  for  telling.  Cognition  and  Instruction,  16, 
475-522. 

Schwartz,  D.  L.,  Bransford,  J.  B.,  &  Sears,  D.  (2005).  Efficiency  and  innovation  in  transfer.  In 
J.  P.  Mestre  (Ed.),  Transfer  of  learning  from  a  modern  multidisciplinary  perspective  (pp. 
1-51).  Greenwich,  CT:  Information  Age  Publishing. 

Schwartz,  D.L.,  Chase,  C.C.,  Oppezzo,  M.A.,  and  Chin,  D.B.  (2011).  Practicing  versus 

inventing  with  contrasting  cases:  the  effects  of  telling  first  on  learning  and  transfer. 
Journal  of  Educational  Psychology,  103,  759-775. 

Schwartz,  D.  L.,  &  Martin,  T.  (2004).  Inventing  to  prepare  for  future  learning:  The  hidden 

efficiency  of  encouraging  original  student  production  in  statistics  instruction.  Cognition 
and  Instruction,  22,  129-184. 

Schwartz,  D.  L.,  Sears,  D.,  &  Chang,  J.  (2008).  Reconsidering  prior  knowledge.  Retrieved  from 
http://aaalab.stanford.edu/papers/Schwartz-Reconsidering-Prior-K.pdf 

Tobias,  S.,  and  Duffy,  T.M.,  eds.,  (2009).  Constructivist  Instruction:  Success  or  Failure?  New 
York:  Routledge. 


30 


Tucker,  J.  S.,  Semmens,  R.  P.,  Sidman,  J.,  Geyer,  A.,  Vaughn,  E.  D.  (2011).  Training  tactical- 
level  planning  skills:  Investigating  problem-centered  and  direct  instruction  approaches 
(Technical  Report  1291).  Arlington,  VA:  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences.  (DTIC  No.  ADA545362) 

U.S.  Department  of  the  Army  (2011).  The  U.S.  Army  learning  concept  for  2015  (TRADOC 
Pamphlet  525-8-2).  Fort  Monroe,  VA:  Author. 

U.S.  Department  of  the  Army  (2009).  Guide  to  army  training  and  education  development: 
Process,  frameworks,  models  and  efficiencies  (TRADOC  Pamphlet  350-70-4).  Fort 
Monroe,  VA:  Author. 

U.S.  Department  of  the  Army  (1999).  Systems  Approach  to  Training  Management,  Process,  and 
Products.  (TRADOC  Reg  350-70,  superseded).  Fort  Monroe,  VA:  Author. 

U.S.  Office  of  Management  and  Budget  (2007).  FEA  Consolidated  Reference  Model  Document, 
Ver  2 . 3 .  Retrieved  from  http  ://www.whitehouse.  gov/sites/ default/ files/ omb/ assets/ 
fea  docs/FEA  CRM  v23  Final  Oct  2007  Revised.pdf 

VandeWalle,  D.  (1997).  Development  and  validation  of  a  work  domain  goal  orientation 
instrument.  Educational  and  Psychological  Measurement,  57,  995-1015. 


31 


32 


Appendix  A 
NCOER  Forms 


1.  NCO  Evaluation  Report 
2.  NCOER  Counseling  and  Support  Form 


A-1 


NCO  EVALUATION  REPORT 

For  Use  of  this  form,  see  AR  623-205;  the  proponent  agency  is  ODCSPER 


SEE  PRIVACY  ACT  STATEMENT 
IN  AR  623-205,  APPENDIX  C 


PART  I  -  ADMINISTRATIVE  DATA 


a.  NAME  (Last,  First,  Middle  Initial) 


b.  SSN 


c.  RANK 


f.  UNIT,  ORG.,  STATION,  ZIP  CODE  OR  APO,  MAJOR  COMMAND 


h.  PERIOD  COVERED 

FROM 

THRU 

YYYY  MM 

YYYY  MM 

i.  RATED 
MONTHS 


j.  NON- 
RATED 
CODES 


k.  NO  OF 
ENCL 


d.  DATE  OF  RANK  e.  PMOSC 


g.  REASON  FOR  SUBMISSION 


I.  RATED  NCO  COPY  Check  one  and  Date) 


1 .  Given  to  NCO 

Date 


m.  PSC 
Initials 


n.  CMD 
CODE 

OT 


o.  PSB 
CODE 


PART  II  -  AUTHENTICATION 


a.  NAME  OF  RATER  (Last,  First,  Middle  Initial) 


SSN 


SIGNATURE 


RANK,  PMOSC/BRANCH,  ORGANIZATION,  DUTY  ASSIGNMENT 


DATE 


b.  NAME  OF  SENIOR  RATER  (Last,  First,  Middle  Initial) 


SSN 


SIGNATURE 


RANK,  PMOSC/BRANCH,  ORGANIZATION,  DUTY  ASSIGNMENT 


DATE 


C.  RATED  NCO:  I  understand  my  signature  does  not  constitute  agreement  or  disagreement  with  the  evaluations  of  the  rater  and  senior  rater.  I  further 
understand  my  signature  verifies  that  the  administrative  data  in  Part  I,  the  rating  officials  in  Part  II,  the  duty  description  to  include  the  counseling  dates  in 
Part  III,  and  the  APFT  and  height/weight  entries  in  Part  IVc  are  correct.  I  have  seen  the  report  completed  through  Part  V,  except  Parts  lid  and  lie.  I  am 
aware  of  the  appeals  process  of  AR  623-205. 


SIGNATURE 


DATE 


d.  NAME  OF  REVIEWER  (Last,  First,  Middle  Initial) 


SSN 


SIGNATURE 


RANK,  PMOSC/BRANCH,  ORGANIZATION,  DUTY  ASSIGNMENT 


DATE 


CONCUR  WITH  RATER  AND  SENIOR  RATER  EVALUATIONS 


NONCONCUR  WITH  RATER  AND/OR  SENIOR  RATER  EVAL  /See  attached  comments) 


PART  III  -  DUTY  DESCRIPTION  (Rater) 


a.  PRINCIPAL  DUTY  TITLE 


b.  DUTY  MOSC 


c.  DAILY  DUTIES  AND  SCOPE  (To  include,  as  appropriate,  people,  equipment,  facilities  and  dollars) 


d.  AREAS  OF  SPECIAL  EMPHASIS 


e.  APPOINTED  DUTIES 


f.  COUNSELING  DATES 


INITIAL 


LATER 


LATER 


LATER 


PART  IV  -  ARMY  VALUES/ATTRIBUTES/SKILLS/ACTIONS  (Rater) 


“NO”.  (Comments  are  mandatory  for  “No”  entries;  optional  for  “Yes”  entries.) 

YES 

NO 

1 .  LOYALTY:  Bears  true  faith  and  allegiance  to  the  U.S.  Constitution,  the  Army,  the  unit,  and  other  soldiers. 

2.  DUTY:  Fulfills  their  obligations 

3.  RESPECT/EO/EEO:  Treats  people  as  they  should  be  treated. 

4.  SELFLESS-SERVICE:  Puts  the  welfare  of  the  nation,  the  Army,  and  subordinates  before  their  own. 

5.  HONOR:  Lives  up  to  all  the  Army  values. 

6.  INTEGRITY:  Does  what  is  right  -  legally  and  morally. 

7.  PERSONAL  COURAGE:  Faces  fear,  danger,  or  adversity  (physical  and  moral). 

Bullet  comments 

V 


A 


Loyalty 

Duty 

Respect 

Selfless-Service 


u 


Honor 

Integrity 

Personal  Courage 


DA  FORM  2166-8,  OCT  2001 


REPLACES  DA  FORM  2166-7,  SEP  87,  WHICH  IS  OBSOLETE 

A-2 


USAPA  VI  .00 


RATED  NCO’S  NAME  (Last,  First.  Middle  Initial) 

SSN 

THRU  DATE 

PART  IV  (Rater)  -  VALUES/NCO  RESPONSIBILITIES 

Specific  Bullet  examples  of  “EXCELLENCE”  or  “NEEDS  IMPROVEMENT"  are  mandatory. 

Specific  Bullet  examples  of  “SUCCESS"  are  optional. 

b.  COMPETENCE 

o  Duty  proficiency;  MOS  competency 
o  Technical  &  tactical;  knowledge,  skills,  and 
abilities 

o  Sound  judgment 

o  Seeking  self-improvement;  always  learning 
o  Accomplishing  tasks  to  the  fullest  capacity; 
committed  to  excellence 

EXCELLENCE  SUCCESS  NEEDS  IMPROVEMENT 

(Exceeds  Std)  (Meets  std)  (Some)  (Much) 


□ 


□ 


□  □ 


PHYSICAL  FITNESS  &  MILITARY  BEARING 
o  Mental  and  physical  toughness 
o  Endurance  and  stamina  to  go  the  distance 
o  Displaying  confidence  and  enthusiasm; 
looks  like  a  soldier 


APFT 


HEIGHT/WEIGHT 


EXCELLENCE 

(Exceeds  Std) 

□ 


SUCCESS 

(Meets  std) 

□ 


NEEDS  IMPROVEMENT 

(Some)  (Much) 

□  □ 


d.  LEADERSHIP 
o  Mission  first 

o  Genuine  concern  for  the  soldiers 
o  Instilling  the  spirit  to  achieve  and  win 
o  Setting  the  example;  Be,  Know,  Do 


EXCELLENCE 

(Exceeds  Std) 

□ 


SUCCESS 

(Meets  std) 

□ 


NEEDS  IMPROVEMENT 

(Some)  (Much) 

□  □ 


e.  TRAINING 

o  Individual  and  team 
o  Mission  focused;  performance  oriented 
o  Teaching  soldiers  how;  common  tasks, 
duty-related  skills 

o  Sharing  knowledge  and  experience  to  fight, 
survive  and  win 

EXCELLENCE  SUCCESS  NEEDS  IMPROVEMENT 

(Exceeds  Std)  (Meets  std)  (Some)  (Much) 


□ 


□ 


□  □ 


f.  RESPONSIBILITY  &  ACCOUNTABILITY 

o  Care  and  Maintenance  of  equipment/facilities 
o  Soldier  and  equipment  safety 
o  Conservation  of  supplies  and  funds 
o  Encouraging  soldiers  to  learn  and  grow 
o  Responsible  for  good,  bad,  right  &  wrong 
EXCELLENCE  SUCCESS  NEEDS  IMPROVEMENT 
(Exceeds  Std)  (Meets  std)  (Some)  (Much) 


□ 


□ 


□  □ 


PART  V  -  OVERALL  PERFORMANCE  AND  POTENTIAL 


a.  RATER.  Overall  potential  for  promotion  and/or 
service  in  positions  of  greater  responsibility. 


AMONG  THE 
BEST 


FULLY 

CAPABLE 


MARGINAL 


□ 


□ 


□ 


b.  RATER.  List  3  positions  in  which  the  rated 
NCO  could  best  serve  the  Army  at  his/her 
current  or  next  higher  grade. 


c.  SENIOR  RATER.  Overall  performance 


e.  SENIOR  RATER  BULLET  COMMENTS 


1  2  3  4  5 

Successful  Fair  Poor 


d.  SENIOR  RATER.  Overall  potential 
for  promotion  and/or  service  in 
positions  of  greater  responsibility. 


1  2  3  4  5 

Superior  Fair  Poor 


DA  FORM  2166-8,  OCT  2001 


REPLACES  DA  FORM  2166-7,  SEP  87,  WHICH  IS  OBSOLETE 

A- 3 


USAPA  VI  .00 


NCOER  COUNSELING  AND  SUPPORT  FORM 

For  use  of  this  form,  see  AR  623-3;  the  proponent  agency  is  DCS,  G-1 . 


FOR  OFFICIAL  USE  ONLY  (FOUO) 
SEE  PRIVACY  ACT  STA  TEMENT 
IN  AR  623-3. 


PART  I  -  ADMINISTRATIVE  DATA 


a.  NAME  (Last,  First,  Middle  Initial) 


b.  SSN 


c.  RANK 


d.  DATE  OF  RANK 


f.  UNIT,  ORG.,  STATION,  ZIP  CODE  OR  APO,  MAJOR  COMMAND 

STATUS  CODE 

1.  RATED  NCO'S  EMAIL  ADDRESS 

m.  UIC 

n.  CMD 

o.  PSB 

(.gov  or  .mil) 

CODE 

CODE 

e.  PMOSC 


PART  II  -  AUTHENTICATION 


a.  NAME  OF  RATER  (Last,  First,  Middle  Initial) 

SSN 

INITIAL 

LATER 

LATER 

LATER 

RANK,  PMOSC/BRANCH,  ORGANIZATION,  DUTY  ASSIGNMENT 

RATER'S  AKO  EMAIL  ADDRESS  (.gov.  or.mil) 

b.  NAME  OF  SENIOR  RATER  (Last,  First,  Middle  Initial) 

SSN 

INITIAL 

LATER 

LATER 

LATER 

RANK,  PMOSC/BRANCH,  ORGANIZATION,  DUTY  ASSIGNMENT 

SENIOR  RATER'S  AKO  EMAIL  ADDRESS  (.gov.  or 
.mil) 

c.  NAME  OF  REVIEWER  (Last,  First,  Middle  Initial) 

SSN 

INITIAL 

LATER 

LATER 

LATER 

RANK,  PMOSC/BRANCH,  ORGANIZATION,  DUTY  ASSIGNMENT 

REVIEWER'S  AKO  EMAIL  ADDRESS  (.gov.  or.mil) 

d.  RATED  NCO's  INITIALS 

INITIAL 

LATER 

LATER 

LATER 

PART  III  -  DUTY  DESCRIPTION  (Rater) 


a.  PRINCIPAL  DUTY  TITLE 


b.  DUTY  MOSC 


c.  DAILY  DUTIES  AND  SCOPE  (To  include,  as  appropriate,  people,  equipment,  facilities  and  dollars) 


d.  AREAS  OF  SPECIAL  EMPHASIS 


e.  APPOINTED  DUTIES 


f.  PHYSICAL  FITNESS  &  MILITARY  BEARING 


APFT 


APFT  DATE 


HEIGHT/WEIGHT 


a.  ARMY  VALUES: 


PART  IV  -  ARMY  VALUES/ATTRIBUTES/SKILLS/ACTIONS  (Rater) 


LOYALTY,  DUTY,  RESPECT/EO/EEO,  SELFLESS-SERVICE,  HONOR,  INTEGRITY,  PERSONAL 


TASK/ACTIONS: 


PERFORMANCE  SUMMARY: 


DA  FORM  2166-8-1,  MAR  2006 


PREVIOUS  EDITIONS  ARE  OBSOLETE. 


Page  1  of  2 

APD  V3.01 


A-4 


RATED  NCO'S  NAME  (Last,  First,  Middle  Initial) 

SSN 

b.  COMPETENCE:  0  Duty  proficiency;  MOS  competency  o  Technical  &  tactical;  knowledge,  skills,  and  abilities 

o  Sound  judgment  o  Seeking  self-improvement;  always  learning 

o  Accomplishing  tasks  to  the  fullest  capacity;  committed  to  excellence 

TASK/ACTIONS: 

PERFORMANCE  SUMMARY: 

c  PHYSICAL  FITNESS  &  MILITARY  BEARING-  0  Mental  and  physical  toughness  o  Endurance  and  stamina  to  go  the  distance 

o  Displaying  confidence  and  enthusiasm;  looks  like  a  soldier 

TASK/ACTIONS: 

PERFORMANCE  SUMMARY: 

d.  LEADERSHIP:  o  Mission  first  o  Genuine  concern  for  soldiers 

o  Instilling  the  spirit  to  achieve  and  win  o  Setting  the  example;  Be,  Know,  Do 

TASK/ACTIONS: 

PERFORMANCE  SUMMARY: 

e.  TRAINING:  0  Individual  and  team  o  Mission  focused;  performance  oriented 

o  Teaching  soldiers  how;  common  tasks,  duty-related  skills  o  Sharing  knowledge  and  experience  to  fight,  survive  and  win 

TASK/ACTIONS: 

PERFORMANCE  SUMMARY: 

f.  RESPONSIBILITY  &  ACCOUNTABILITY:  0  Dare  and  maintenance  of  equipment/facilities  o  Soldier  and  equipment  safety 

o  Conservation  of  supplies  and  funds  o  Encouraging  soldiers  to  learn  and  grow 

o  Responsible  for  good,  bad,  right  &  wrong 

TASK/ACTIONS: 

PERFORMANCE  SUMMARY: 

Page  2  of  2 


APD  V3.01 


DA  FORM  2166-8-1,  MAR  2006 


A-5 


A-6 


Appendix  B 

Demographic  Questionnaire 


B-l 


Background  Information 


1 .  Rank  (circle  one) 

SGT  SSG  SFC 

Years 

Months 

2.  Time  in  Service  (TIS) 

3.  Time  in  Grade  (TIG) 

4.  Civilian  Education  Level 
(circle  highest  level) 

Non  HSG 

GED 

HS  Diploma 

Some 
College 
(no  degree) 

Associate 

Degree 

Bachelor 

Degree 

Graduate 

Work 

Master 

Degree 

YES 

NO 

5.  Received  recognition  for  writing  (HS  or  later) 

6.  Completed  college  level  writing  class 

7.  Regularly  conduct  quarterly  counseling 

8.  Regularly  use  the  NCOER  support  form 

NCOER  Experience 


9.  Based  on  your  personal  experience,  when  you  were  the  RATER  for  an  NCOER  -  - 

CD 

Platoon 

Level 

Company 

Level 

Someone 
above  me 
(unknown) 

a.  Who  completed  the  administrative  section  (Part  1  &  II)? 

O 

o 

o 

o 

b.  Who  developed  the  bullet  comments  (Part  IV)? 

o 

o 

o 

o 

c.  Who  determined  the  ratings  (Excellence,  Success,  Nl)? 

o 

o 

o 

o 

10.  Mark  the  appropriate  number. 

- 

2  or  3 

4  or  5 

More  than  5 

a.  How  many  NCOERs  have  you  been  the  RATER  for? 

o 

o 

o 

o 

b.  How  many  NCOERs  have  you  received? 

o 

o 

o 

o 

1 1 .  Write  a  single  sentence  about  one  of  your  significant  leadership  accomplishments. 
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General  Instructions:  These  questions  will  not  be  used  for  academic  evaluations  in 
the  Advanced  Leaders  Course.  They  will  only  be  used  to  assess  your  knowledge  and 
skills  on  completing  NCOERs.  If  you  are  uncertain  of  the  correct  answer,  please 
select  the  response  “I  am  unsure  of  the  answer”.  Record  only  those  answers  that 
you  believe  are  correct. 

GENERAL  SITUATION:  For  this  scenario  you  will  play  the  role  of  SSG  Thatwood  B. 
Me,  the  Squad  Leader  for  the  2nd  Squad,  3rd  Platoon,  A  Company,  1st  Battalion  (CAB), 
56th  Infantry  Regiment  assigned  to  the  2nd  Brigade  (HBCT),  21st  Infantry  Division.  1-56 
INF  is  a  FORSCOM  unit  posted  to  Camp  Swampy,  Georgia.  The  1-56  INF  was 
redeployed  from  Iraq  in  November  2010  and  after  a  period  of  recovery  followed  by 
assignment  turbulence,  the  unit  is  now  focused  on  training  for  full  spectrum  operations. 

Your  squad  has  gone  through  a  period  of  individual  skills  training  that  included  Expert 
Infantry  Badge  (EIB)  training  and  testing.  You  are  now  transitioning  to  small  unit 
collective  training,  gunnery,  and  field  training  that  will  culminate  with  a  training 
deployment  (in  three  and  one-half  months)  to  the  National  Training  Center  at  Fort  Irwin, 
CA  against  the  opposing  force  (OPFOR)  regiment,  a  full  spectrum  operations  rotation. 

For  this  problem  scenario,  today  is  Friday  21  October  2011.  You  are  preparing  for 
your  final  counseling  session  with  SGT  Public  and  completing  his  NCOER. 

SPECIAL  SITUATION:  Sergeant  John  Q.  Public,  your  Bravo  Fire  Team  Leader  (TL)  is 
being  reassigned;  departing  28  October  201 1 .  SGT  Public  has  been  the  TL  for  10 
months,  assuming  duties  upon  return  from  convalescent  leave.  You  took  over  the 
Squad  4  months  ago.  Your  predecessor  gave  SGT  Public  his  first  NCOER,  a  “Change 
of  Rater”  with  an  end  date  of  30  June  201 1 . 

You  will  be  provided  the  working  file  for  his  NCOER  that  will  assist  you  in  completing 
this  rating  prior  to  his  departure  for  his  next  duty  station,  Fort  Hood,  TX.  The  working 
file  will  contain  the  following  items  for  your  review  and  use: 

•  A  partially  completed  NCO  Evaluation  Report  (DA  Form  2166-8) 

•  A  partially  completed  NCOER  Counseling  &  Support  Form  (DA  Form  2166-8-1 ), 
initial  counseling  on  14  July  201 1 

•  A  Developmental  Counseling  Form  (DA  Form  4856),  from  26  Aug  201 1 

•  SGT  Public’s  file  with  miscellaneous  orders,  records,  and  notes: 

o  Appointment  orders,  Company  Energy  and  Water  Management  NCO 
o  Emergency  Leave  Request  and  Extension  (28  Aug  -  2  Oct) 
o  Memo:  A  Co.  EIB  testing  results  (15  Sep) 

o  Memo:  Installation  Energy  and  Water  Conservation  Inspection  (22  Sep) 
o  APFT  Scorecard  with  data  for  01/07/11, 03/18/11,  and  09/23/11 
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You  may  use  the  materials  provided,  AR  623-3,  and  DA  Pam  623-3  to  answer  questions 
that  follow.  Record  your  answers  on  form  provided.  Turn  in  all  materials  when 
finished. 

There  are  rules  and  principles  that  guide  writing  bullets  for  NCOERs.  Bullets  should 
provide  the  reader  with  details  and  examples  of  the  rated  NCO’s  performance,  as  well 
as  providing  support  and  justification  for  the  selected  rating.  From  the  list  below  identify 
the  rules  and  principles  that  should  be  used  to  create  bullets  from  your  observations 
and  counseling  records. 

For  items  1  through  6  those  techniques  that  should  be  used,  enter  a  “Y”  for  YES; 

for  those  techniques  that  should  not  be  used,  enter  an  “N”  for  NO; 

for  items  that  you  are  unsure  of  or  do  not  know,  enter  a  “U”  for  UNSURE. 

_ 1 .  Only  include  bolded,  underlined,  or  italicized  text  for  special  emphasis. 

_ 2.  Separate  bullets  by  double-spacing. 

_ 3.  Begin  with  a  small  letter  “o”  to  designate  the  start  of  a  bullet. 

_ 4.  Only  complete  and  grammatically  correct  sentence  bullets  may  use  two  lines. 

_ 5.  Reuse  bullets  for  more  than  one  value  or  responsibility,  when  appropriate. 

_ 6.  Phrase  bullet  may  be  used  but  are  limited  to  one  line  of  text. 


For  items  7  through  14  refer  to  SGT  Public’s  counseling  files. 

7.  What  are  the  correct  entries  for  boxes  “i.  RATED  MONTFIS”  and  “j.  NON-RATED 
CODES”  in  “PART  I  -  ADMINISTRATIVE  DATA”  of  the  NCO  EVALUATION  REPORT? 

A.  i.  RATED  MONTHS  _4_ ;  j.  NON-RATED  CODES _ (leave  blank;  make  no  entry) 

B.  i.  RATED  MONTHS  _4_;  j.  NON-RATED  CODES  E 

C.  i.  RATED  MONTHS  _3_;  j.  NON-RATED  CODES _ (leave  blank;  make  no  entry) 

D.  i.  RATED  MONTHS  _3_;  j-  NON-RATED  CODES  E 

E.  i.  RATED  MONTHS  _3_;  j-  NON-RATED  CODES  Z 

F.  I  am  unsure  of  the  answer. 
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8.  While  reviewing  Part  III  c.  DAILY  DUTIES  AND  SCOPE  of  SGT  Public’s  NCOER, 
you  determine  that  it  is  not  fully  in  compliance  with  the  guidance  and  instructions  in  DA 
Pam  623-3,  Evaluation  Reporting  System.  Select  from  the  item  below  that  should  be 
included  in  this  section  of  the  NCOER. 

A.  Important  routine  duties  and  responsibilities. 

B.  Number  of  Soldiers  supervised. 

C.  Equipment  and  facilities  for  which  the  rated  NCO  is  responsible  and  their  dollar 
value. 

D.  All  the  above. 

E.  I  am  unsure  of  the  answer. 


9.  In  reviewing  SGT  Public’s  records  you  determine  that  a  rating  of  EXCELLENCE  is 
appropriate  for  item  c.  in  Part  IV.  PHYSICAL  FITNESS  AND  MILITARY  BEARING. 
Based  on  the  information  available,  write  a  bullet  that  fully  supports  the  EXCELLENCE 
rating. 

(Write  [print]  your  bullet  on  the  answer  sheet  in  the  space  provided.) 


10.  Examples  of  unit  achievements  or  performance  are  sometimes  used  in  Part  IV.  d. 
LEADERSHIP  and/or  e.  TRAINING.  During  the  rating  period,  three  (3)  members  of 
Bravo  Fire  Team  passed  all  requirements  and  were  awarded  the  Expert  Infantry  Badge 
(EIB).  How  could  you  reflect  this  information  on  SGT  Public’s  NCOER? 

A.  Testing  and  award  occurred  during  the  period  that  SGT  Public  was  on  leave.  It  was 
non-rated  time;  no  comments  can  be  made  on  the  NCOER. 

B.  SGT  Public  met  the  standard.  No  narrative  bullet  is  required  for  SUCCESS  ratings. 

C.  The  Bravo  Team  Soldiers  exceeded  the  established  standard,  but  SGT  Public  was 
not  present,  did  not  finish  their  training,  and  did  not  lead  them  through  testing.  No  bullet 
comment  is  appropriate. 

D.  A  bullet  comment  in  LEADERSHIP  or  TRAINING  could  state,  “o  prepared  all  3 
subordinates  for  the  Expert  Infantry  Badge,  all  were  tested  and  awarded  the  EIB”. 

E.  I  am  unsure  of  the  answer. 
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1 1 .  You  determine  that  SGT  Public’s  performance  in  RESPONSIBILITY  & 
ACCOUNTABILITY  (Part  IV.  f.)  should  receive  an  EXCELLENCE  rating  in  part  for  the 
Alpha  Company’s  Energy  and  Water  Management  results  for  the  Installation  Inspection 
Based  on  the  information  available,  write  a  bullet  that  fully  supports  the  EXCELLENCE 
rating. 

(Write  [print]  your  bullet  on  the  answer  sheet  in  the  space  provided.) 


12.  You  determine  that  SGT  Public’s  performance  in  TRAINING  (Part  IV.  e.)  should 
receive  an  EXCELLENCE  rating.  Select  the  bullet  that  best  supports  this  rating. 

A.  o  conducted  daily  additional  PT  and  Warrior  Skills  training  preparing  his  Soldiers  for 
EIB  testing;  all  his  Soldiers  qualified  for  the  EIB 

B.  o  improved  marksmanship  training  assisting  all  his  Soldiers  to  qualify  Sharpshooter 
or  better 

C.  o  assisted  Soldiers  and  superiors  to  improve  APFT  scores,  entire  Fire  Team  and 
Platoon  Leader  met  or  exceeded  company  APFT  standards 

D.  o  always  assigned  difficult  classes;  one  of  the  best  trainers  in  the  platoon 

E.  I  am  unsure  of  the  answer. 


13.  You  want  to  provide  some  strong  bullets  to  highlight  SGT  Public’s  demonstrated 
LEADERSHIP  (Part  VI.  d.)  during  the  rating  period.  The  bullet  that  you  write  needs  to 
provide  substantive,  quantifiable  results,  or  performance  showing  how  he  set  the 
example  (Be,  Know,  Do).  Select  the  statement  below  that  provides  the  best  input  for  a 
bullet. 

A.  He  scored  272  on  the  APFT  and  obtained  90  points  or  more  in  all  events. 

B.  He  routinely  led  platoon  PT  and  conducted  extra  training  and  PT. 

C.  He  now  meets  height  and  weight  standards  and  has  no  physical  profile. 

D.  He  conducted  regular  counseling  sessions  with  subordinates  to  correct  deficiencies 
and  shortcomings. 

E.  I  am  unsure  of  the  answer. 
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14.  In  Part  IV  section  a.  ARMY  VALUES  of  the  NCOER,  YES  is  indicated  all  the  boxes. 
It  is  apparent  from  the  file  that  SGT  Public  has  a  strong  sense  of  duty.  Write  a  bullet 
that  provides  a  quantitative  or  measurable  example  of  SGT  Public’s  fulfillment  of 
assigned  duties  or  obligations. 

(Write  [print]  your  bullet  on  the  answer  sheet  in  the  space  provided.) 


Items  15  through  17  cover  basic  rules  and  principles  of  the  NCOER  Evaluation 
Reporting  System.  You  may  refer  to  AR  623-3  and  DA  Pam  623-3,  if  required. 


15.  When  selecting  value  ratings,  SUCCESS  indicates  that  the  rated  NCO  meets  the 
standard  and  is  fully  competitive  for  promotion  and  schooling.  Your  goal  in  counseling 
is  to  encourage  and  guide  your  subordinates  to  attain  and  maintain  this  level.  Identify 
the  bullet  below  that  would  not  support  or  justify  a  SUCCESS  rating. 

A.  o  sustains  a  comprehensive  cross-training  program  for  Soldiers  in  his  team 

B.  o  continually  seeking  improvement,  completed  one  college  level  course 

C.  o  encouraged  all  Soldiers  to  meet  company  standards  on  APFT 

D.  o  led  his  unit  to  qualify  as  Marksman  during  semi-annual  qualification 

E.  o  led  Soldiers  to  complete  the  company  5-mile  run 

F.  I  am  unsure  of  the  answer. 


16.  Ratings  of  NEEDS  IMPROVEMENT  indicate  that  the  rated  NCO  has  missed  or 
failed  to  meet  standards  for  a  specific  Army  value  or  responsibility.  A  rating  of  NEEDS 
IMPROVEMENT  should  not  be  a  surprise  to  the  rated  NCO.  If  the  NCO  received  an 
adverse  administrative  action  or  an  Article  15  during  the  rating  period,  indicate  how  to 
include  this  information  in  an  NCOER  bullet. 

A.  For  an  Article  15,  state  “Article  15”  and  include  the  charge  and  specification 

B.  For  an  Article  15,  state  “Article  15”  and  include  the  punishment  adjudged 

C.  For  an  adverse  action,  state  the  level  of  command  taking  the  action  and  results 

D.  State  the  NCO’s  conduct  or  performance  that  caused  the  action  or  punishment 

E.  I  am  unsure  of  the  answer. 
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1 7.  In  Part  IV  section  a.  ARMY  VALUES  of  the  NCOER  the  rater  is  required  to  provide 
his  evaluation  of  the  rated  NCO’s  adherence  to  Army  values  through  his  attributes, 
skills,  and  actions.  An  “X”  must  be  placed  in  the  appropriate  rating  box  in  each  section. 
There  is  limited  space  provided  for  bullets.  Which  rule  or  guide  below  is  not 
appropriate  or  correct  for  the  narrative  bullets  used  in  the  section? 

A.  Quantitative  or  substantial  bullets  are  to  be  used  to  explain  an  area  where  the  rated 
NCO  is  particularly  strong  or  where  he  is  below  his  peers. 

B.  Specific  bullets  are  mandatory  for  all  “NO”  entries. 

C.  Bullets  are  not  mandatory  for  “YES”  entries. 

D.  Bullets  may  be  reused  on  the  back  in  Part  IV.  sections  b.  —  f. ,  where  appropriate. 

E.  I  am  unsure  of  the  answer. 
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ALC  NCOER  Diagnostic  Test  2 

(Ver.  005.0,  13  Sept  2011) 


General  Instructions:  These  questions  will  not  be  used  for  academic  evaluations  in 
the  Advanced  Leaders  Course.  They  will  only  be  used  to  assess  your  knowledge  and 
skills  on  completing  NCOERs.  If  you  are  uncertain  of  the  correct  answer,  please 
select  the  response  “I  am  unsure  of  the  answer”.  Record  only  those  answers  that 
you  believe  are  correct. 

GENERAL  SITUATION:  For  this  scenario  you  will  play  the  role  of  SSG  None  Better, 
the  Squad  Leader  for  the  2nd  Squad,  3rd  Platoon,  A  Company,  1st  Battalion  (CAB),  71th 
Infantry  Regiment  assigned  to  the  2nd  Brigade  (HBCT),  21st  Infantry  Division.  1-71  INF 
is  a  FORSCOM  unit  posted  to  Camp  Swampy,  Georgia.  The  1-71  INF  was  redeployed 
from  Iraq  in  November  2010  and  after  a  period  of  recovery  followed  by  assignment 
turbulence,  the  unit  is  now  focused  on  training  for  full  spectrum  operations. 

Your  squad  has  gone  through  a  period  of  individual  skills  training.  You  are  now 
transitioning  to  small  unit  collective  training,  and  field  training  that  will  culminate  with  a 
training  deployment  (in  three  and  one-half  months)  to  the  Joint  Readiness  Training 
Center  at  Fort  Polk,  LA  against  the  opposing  force  (OPFOR)  regiment,  a  full  spectrum 
operations  rotation. 

For  this  problem  scenario,  today  is  Friday  21  October  2011.  You  are  preparing  for 
your  final  counseling  session  with  SGT  Makit  and  completing  his  NCOER. 

SPECIAL  SITUATION:  You  are  being  reassigned  with  a  departure  of  28  October  201 1 . 
You  must  give  Sergeant  Hardly  W.  Makit  his  first  NCOER,  which  will  be  a  “Change  of 
Rater”.  SGT  Makit  has  been  your  Bravo  Fire  Team  Leader  (TL)  since  January  201 1 , 
assuming  duties  just  after  his  promotion  from  Specialist  to  Sergeant. 

You  will  be  provided  the  working  file  for  his  NCOER  that  will  assist  you  in  completing 
this  rating  prior  to  your  departure.  The  working  file  will  contain  the  following  items  for 
your  review  and  use: 

•  A  partially  completed  NCO  Evaluation  Report  (DA  Form  2166-8) 

•  A  partially  completed  NCOER  Counseling  &  Support  Form  (DA  Form  2166-8-1 ), 
initial  counseling  on  14  Jan  201 1 

•  3  Developmental  Counseling  Forms  (DA  Form  4856),  dated:  18  Apr.,  22  July, 
and  22  Sept.  201 1 

•  SGT  Public’s  file  with  miscellaneous  orders,  records,  and  notes: 

o  Appointment  orders,  Company  Representative  to  Highland  High  School 
o  2  Memos:  Pertaining  to  equipment  loss  (22  Sep) 
o  APFT  Scorecard  with  data  for  1 1  /1 5/1 0,  05/1 8/1 1 ,  and  10/17/11 

You  may  use  the  materials  provided,  AR  623-3,  and  DA  Pam  623-3  to  answer  questions 
that  follow.  Record  your  answers  on  form  provided.  Turn  in  all  materials  when 
finished. 

There  are  rules  and  principles  that  guide  writing  bullets  for  NCOERs.  Bullets  should 
provide  the  reader  with  details  and  examples  of  the  rated  NCO’s  performance,  as  well 
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as  providing  support  and  justification  for  the  selected  rating.  From  the  list  below  identify 
the  rules  and  principles  that  should  be  used  to  create  bullets  from  your  observations 
and  counseling  records. 

For  items  1  through  6  those  techniques  that  should  be  used,  enter  a  “Y”  for  YES; 

for  those  techniques  that  should  not  be  used,  enter  an  “N”  for  NO; 

for  items  that  you  are  unsure  of  or  do  not  know,  enter  a  “U”  for  UNSURE. 

_ 1 .  Phrase  bullet  may  be  used  but  are  limited  to  one  line  of  text. 

_ 2.  Only  complete  and  grammatically  correct  sentence  bullets  may  use  two  lines. 

_ 3.  Only  include  bolded,  underlined,  or  italicized  text  for  special  emphasis. 

_ 4.  Begin  with  a  small  letter  “o”  to  designate  the  start  of  a  bullet. 

_ 5.  Reuse  bullets  for  more  than  one  value  or  responsibility,  when  appropriate. 

_ 6.  Separate  bullets  by  double-spacing. 

For  items  7  through  14  refer  to  SGT  Makit’s  counseling  files. 

7.  What  are  the  correct  entries  for  boxes  “i.  RATED  MONTHS”  and  “j.  NON-RATED 
CODES”  in  “PART  I  -  ADMINISTRATIVE  DATA”  of  the  NCO  EVALUATION  REPORT? 

A.  i.  RATED  MONTHS  _7_;  j.  NON-RATED  CODES _ (leave  blank;  make  no  entry) 

B.  i.  RATED  MONTHS  _7_;  j.  NON-RATED  CODES  E 

C.  i.  RATED  MONTHS  _7_;  j.  NON-RATED  CODES  E 

D.  i.  RATED  MONTHS  _9_;  j.  NON-RATED  CODES  _Z_ 

E.  i.  RATED  MONTHS  _9_;  j.  NON-RATED  CODES _ (leave  blank;  make  no  entry) 

F.  I  am  unsure  of  the  answer. 
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8.  Reviewing  Part  III  c.  DAILY  DUTIES  AND  SCOPE  of  SGT  Makit’s  NCOER  and 
consider  the  guidance  and  instructions  in  DA  Pam  623-3,  Evaluation  Reporting  System. 
Select  the  item  below  that  best  describes  this  section  of  the  NCOER. 


A.  Daily  duties  and  scope  appear  to  be  complete. 

B.  Important  routine  duties  and  responsibilities  seem  to  be  omitted. 

C.  Number  of  Soldiers  supervised  was  omitted. 

D.  Equipment  and  facilities  for  which  the  rated  NCO  is  responsible  and  their  dollar 
value  was  omitted. 

E.  I  am  unsure  of  the  answer. 

9.  In  reviewing  SGT  Makit’s  records  you  determine  that  a  rating  of  NO  is  appropriate  for 
item  a.  6.  in  Part  IV.  ARMY  VALUES/ATTRIBUTES/SKILLS/ACTIONS.  Based  on  the 
information  available,  write  a  bullet  that  fully  supports  the  NO  rating. 

(Write  [print]  your  bullet  on  the  answer  sheet  in  the  space  provided.) 


10.  During  the  rating  period,  the  Battalion  Commander  received  very  complementary 
comments  from  the  District  School  Board  and  City  Council  for  a  static  display  and  color 
guard  for  which  SGT  Makit  was  the  NCOIC.  Based  on  the  information  available,  write  a 
bullet  that  would  indicate  performance  that  “Exceeds  Standard”  in  Part  IV. b., 
CCOMPETENCE. 


(Write  [print]  your  bullet  on  the  answer  sheet  in  the  space  provided.) 


1 1 .  During  the  rating  period,  SGT  Makit  had  a  record  of  being  late  for  early  formations. 
Based  on  the  information  available,  write  a  bullet  that  would  support  a  NEEDS 
IMPROVEMENT  rating  in  Part  Vl.d.,  LEADERSHIP. 


(Write  [print]  your  bullet  on  the  answer  sheet  in  the  space  provided.) 
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12.  During  the  rating  period,  SGT  Makit’s  daily  appearance  was  below  that  expected  of 
an  NCO.  Based  on  the  information  available,  write  a  bullet  that  would  support  a  NEEDS 
IMPROVEMENT  rating  in  Part  Vl.d.,  LEADERSHIP. 


(Write  [print]  your  bullet  on  the  answer  sheet  in  the  space  provided.) 


13.  During  the  rating  period,  SGT  Makit’s  personal  and  fire  team’s  APFT  scores 
continued  to  drop.  Based  on  the  information  available,  write  a  bullet  that  would  support 
a  NEEDS  IMPROVEMENT  rating  in  Part  Vl.c.,  PHYSICAL  FITNESS  AND  MILITARY 
BEARING. 


(Write  [print]  your  bullet  on  the  answer  sheet  in  the  space  provided.) 


14.  In  Part  V  section  a.  how  would  you  rate  SGT  Makit’s  “Overall  potential  for 
promotion  and/or  service  in  positions  of  greater  responsibility.”? 

A.  Among  the  best 

B.  Fully  capable 

C.  Marginal 

D.  Leave  the  rating  blank;  overall  potential  is  not  rated  on  the  initial  NCOER. 

E.  I  am  unsure  of  the  answer. 
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Item  15  relates  to  basic  rules  and  principles  of  the  NCOER  Evaluation  Reporting 
System.  You  may  refer  to  AR  623-3  and  DA  Pam  623-3,  if  required. 

1 5.  In  Part  IV  section  a.  ARMY  VALUES  of  the  NCOER  the  rater  is  required  to  provide 
his  evaluation  of  the  rated  NCO’s  adherence  to  Army  values  through  his  attributes, 
skills,  and  actions.  An  “X”  must  be  placed  in  the  appropriate  rating  box  in  each  section. 
There  is  limited  space  provided  for  bullets.  Which  rule  or  guide  below  is  not 
appropriate  or  correct  for  the  narrative  bullets  used  in  the  section? 

A.  Quantitative  or  substantial  bullets  are  to  be  used  to  explain  an  area  where  the  rated 
NCO  is  particularly  strong  or  where  he  is  below  his  peers. 

B.  Specific  bullets  are  mandatory  for  all  “NO”  entries. 

C.  Bullets  are  not  mandatory  for  “YES”  entries. 

D.  Bullets  may  be  reused  on  the  back  in  Part  IV.  sections  b.  —  f. ,  where  appropriate. 

E.  I  am  unsure  of  the  answer. 
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Post  Training  Questionnaire 


Please  fill-in  the  bubble  to  indicate  your  level  of  understanding 
“Prior  to  Today’s  instruction”  and  “Now”,  following  the  instruction. 


Need  some 
explanation 

Understand 

Somewhat 

Got  it 

Understand 
Very  Well 

Could 
Teach 
This  to 
Others 

1.  Significance  of  conducting 
meaningful  “counseling” 

Prior 

to 

Today 

O 

O 

o 

O 

O 

and  documenting  the 
results. 

Now 

O 

O 

o 

O 

O 

2.  Significance  of  using  the 
“NCOER  support  form”  to 
fully  document 
accomplishments  and 
performance. 

Prior 

to 

Today 

o 

o 

o 

o 

o 

Now 

o 

o 

o 

o 

o 

3.  Importance  of  including 
meaningful  bullets  on  the 

Prior 

to 

Today 

o 

o 

o 

o 

o 

NCOER  that  accurately 
document  performance. 

Now 

o 

o 

o 

o 

o 

4.  Significance  of  supporting 
documentation  in 
developing  ratings  and 
meaningful  bullets  on  the 
NCOER. 

Prior 

to 

Today 

o 

o 

o 

o 

o 

Now 

o 

o 

o 

o 

o 

Please  fill-in  the  bubble  to  indicate  your  feelings  toward  the  instruction  you  received 
today. 


Not  Very 
Useful 

Don’t 

Know 

Useful 

Very 

Useful 

1.  Hand-outs  and  examples  provided  to  support  the 
instruction. 

O 

O 

O 

O 

2.  Hand-outs  for  future  reference. 

O 

O 

o 

o 

3.  Guided  use  of  publications  (regulations  and 
pamphlets)  throughout  the  instruction. 

o 

o 

o 

o 
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Please  fill-in  the  bubble  to  indicate  your  feelings  toward  the  instruction  you  received 
today. 


Strongly 

Disagree 

Disagree 

Neither 
Agree  nor 
Disagree 

Agree 

Strongly 

Agree 

I.The  instruction  gave  me  a  much 
better  understanding  of  the  NCOER 
process. 

O 

O 

O 

O 

O 

2.  The  instruction  prepared  me  to 
perform  my  duties  as  a  rater. 

O 

o 

O 

o 

O 

3. The  instruction  prepared  me  to 
develop  more  meaningful  and 
accurate  NCOERs. 

o 

o 

o 

o 

o 

4. The  time  devoted  to  explaining 
concepts  was  adequate. 

o 

o 

o 

o 

o 

5. The  instruction  covered  issues  and 
nuances  in  the  NCOER  process  that 
were  very  helpful. 

o 

o 

o 

o 

o 

6. The  instruction  provided  valuable 
insights  on  how  to  effectively  develop 
appropriate  bullets  for  an  NCOER. 

o 

o 

o 

o 

o 

7. The  instruction  improved  my  ability  to 
write  an  NCOER  that  accurately 
represents  the  NCO’s  performance. 

o 

o 

o 

o 

o 

8.  The  instruction  provided  me  a  better 
appreciation  for  accessing  and  using 
Army  publications. 

o 

o 

o 

o 

o 

9. The  instruction  assisted  me  in 
knowing  what  1  need  to  do  to  receive 
an  “Excellent”  rating. 

o 

o 

o 

o 

o 

10.  The  instruction  has  motivated  me  to 
ensure  my  subordinates  receive 
accurate  NCOERs. 

o 

o 

o 

o 

o 

11.  The  instruction  has  motivated  me  to 
ensure  that  my  NCOER  accurately 

o 

o 

o 

o 

o 

reflects  my  performance. 


12.  What  aspects  of  the  instruction  were  most  beneficial?  Please  explain. 


13.  What  aspects  of  the  instruction  were  least  beneficial?  Please  explain. 


14.  What  additional  topics  should  be  included  in  the  NCOER  instruction? 
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